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ABSTRACT 


The  quality  of  software  management  has  an  affect  on  the  degree  of  success  or 
failure  of  a  software  development  program,  this  statement  has  been  argued  successfully 
by  Martin  J.  Machniak  in  his  thesis  Development  of  a  Quality  Management  Metric 
(QMM)  Measuring  Software  Program  Management  Quality.  The  QMM  metrics  can  be 
used  both  to  characterize  the  quality  of  software  management  and  provide  a  template  for 
improving  software  management  performance. 

Technical  Perfonnance  Measurement  (TPM)  in  the  most  basic  form  is  a  plan  of 
expected  technical  achievement  in  which  the  actual  progress  is  compared  with  periodic 
measurements.  However,  the  difference  between  the  plan  and  the  actual  measures  is  a 
technical  variance  which  can  be  considered  good  or  bad,  depending  on  the  level  of 
tolerance  given  in  the  requirements.  TPM  is  breaking  new  ground  in  the  development  of 
various  techniques  for  TPM  where  planning  is  integrated  with  cost,  schedule,  and 
program  impact  assessment. 

The  author  administered  the  QMM  questionnaire  to  measure  the  perceptions  of 
program  managers  that  have  the  responsibility  for  software  development  programs  within 
the  U.S.  Anny.  The  author  then  gathered  TPM  data  using  an  informal  verification  and 
validation  of  the  same  programs  used  for  the  QMM  questionnaire,  and  compared  the 
results  and  found  an  inconclusive  correlation  between  them. 
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I.  INTRODUCTION  AND  BACKGROUND 


A.  PROBLEM 

The  U.S.  Army  is  faced  with  the  challenge  of  what  are  the  best  possible 
management  tools  to  use  for  developing  a  more  responsive,  and  a  more  dominate  combat 
force  to  meet  today’s  needs  and  all  future  threats. 

The  U.S.  Anny  is  presently  developing  an  advanced  family  of  networked  air- 
based  and  ground-based  vehicles  that  are  used  in  maneuver;  maneuver  support;  and 
sustain  program  systems  including  manned  and  unmanned  platforms. 

These  systems  are  networked  by  a  Command,  Control,  Communications, 
Computers,  Intelligence,  Surveillance,  and  Reconnaissance  (C4ISR)  architecture  which 
includes  network  communications,  network  operations,  sensors,  battle  command  systems, 
manned/unmanned  reconnaissance  and  surveillance  capabilities  to  enable  levels  of 
situational  understanding  and  synchronized  operations.  The  vehicle  platfonns  are  a 
fraction  of  the  weight  of  the  current  weapon  systems,  and  are  just  as  lethal  and 
survivable. 

The  lightweight  and  smaller  sizes  are  critical  to  meeting  the  Army’s  future  force 
deploy-ability  goal,  of  transporting  vehicles  using  C-130  aircraft.  The  technical 
challenges  are  unprecedented  plus  the  time  constraints  are  formidable  for  this  program. 

One  of  the  major  technical  challenges  is  the  development  of  a  first-of-a-kind 
communication  network.  This  endeavor  includes  developing  data  for  18  advanced 
systems,  with  53  critical  technologies,  employing  157  complementary  systems,  and  34 
million  estimated  lines  of  software  code. 

Traditionally  the  U.S.  Army  usually  allows  only  5.5  years  for  development  of  a 
single  major  system  (between  program  starts  and  the  production  decision).  The  programs 
are  tasked  to  compress  development  time  even  though  this  U.S.  Army  system  of  systems 
is  comprised  of  several  systems  including:  the  network;  an  Abrams  Tank  replacement; 
Bradley  fighting  vehicles  replacements,  and  a  Crusader  replacement. 
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The  U.S.  Army  has  been  given  the  challenge  to  proceed  with  the  strategy  of  using 
a  timetable  where  over  75  percent  of  the  critical  technologies  are  immature.  If  the  U.S. 
Anny  assumes  everything  goes  as  planned,  the  program  will  begin  production  most  likely 
before  all  of  its  systems  have  been  demonstrated.  This  is  the  kind  of  strategy  the  Army 
plans  on  going  forward  with  for  production  and  fielding  of  its  systems. 

The  U.S.  Army  is  now  in  the  System  Development  and  Demonstration  (SDD) 
phase.  The  U.S.  Army’s  acquisition  program  was  approved  by  the  Defense  Acquisition 
Board  in  May  2003.  Also  there  has  been  designated  a  joint  Services  program  with  the 
Army  and  Marine  Joint  Program  Office.  On  July  22,  2004,  Army  officials  announced 
plans  to  accelerate  the  delivery  of  selected  vehicles  to  the  current  force.  The  plan  expands 
the  scope  of  the  program’s  SDD  phase  by  adding  four  discrete  “spin  outs”  of  capabilities 
at  two  year  increments  for  the  current  forces. 

Spin  out  one  will  begin  fielding  in  2008  and  consist  of  prototypes  fielded  to  the 
Evaluation  Brigade  Combat  Team  (EBCT).  Following  successful  evaluation,  production 
and  fielding  of  spin  out  one  will  commence  in  2010.  This  process  will  be  repeated  for 
each  successive  spin  out.  By  2014,  the  EBCT  will  be  equipped  with  all  new  core  systems. 
Other  Brigade  Combat  Teams  will  have  selected  embedded  new  capability. 

This  is  the  Army’s  strategy  for  the  main  modernization  program  in  the  21st 
century.  It  will  ensure  that  the  Army  retains  the  combat  advantage  in  critical  capabilities 
plus  having  net-centricity,  mobility,  and  a  more  efficient  use  of  material  and  personnel. 
When  fielded  to  the  force,  the  U.S.  Anny  will  have  replaced  40  year  old  equipment 
designed  to  win  against  Cold  War  enemies.  This  effort  will  benefit  the  Anny,  Marine 
Corps,  and  Special  Operations  Forces,  and  the  Nation  as  a  whole. 

Since  software  development  is  a  major  part  of  this  new  U.S.  Army  system  of 
systems,  it  is  imperative  that  the  software  development  be  managed  effectively  in  order  to 
assure  that  the  Army’s  strategy  is  successfully  implemented.  Effective  management  of 
the  software  development,  in  turn,  requires  that  the  requirements  for  effective 
management  be  understood,  measured  and  monitored. 
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B.  SOLUTION 

The  author  has  in  the  following  sections,  of  this  thesis  examined  U.S.  Army 
software  programs  to  determine  how  well  Quality  Management  Metrics  (QMM)  correlate 
to  Technical  Performance  Measurement  (TPM).  The  author  administered  QMM 
questionnaire  surveys  to  software  Program  Managers  (PM)  in  software  acquisition,  and 
compared  data  from  TPMs  within  the  same  programs.  The  thinking  behind  this  research 
was  to  explore  the  data  of  TPM  and  QMM  to  see  if  there  is  a  good  working  relationship 
between  the  metrics  of  each  and  how  a  software  program  might  benefit  the  U.S.  Army  in 
managing  these  enormous  developmental  projects  that  can  have  tremendous  political 
ramification  and  unwanted  consequences.  However,  what  the  author  did  not  do  and  has 
left  for  future  research  work  was  to  address  the  relationship  between  Earned  Valued  (EV) 
and  QMM. 

C.  RESEARCH  QUESTION 

This  thesis  focuses  on  answering  the  following  question: 

1.  How  well  does  the  quality  management  metrics  (QMM)  correlate  to  the 
technical  performance  measurement  (TPM)? 

D.  SCOPE,  LIMITATION,  AND  ASSUMPTIONS 

This  thesis  describes  how  quality  management  metrics  (QMM)  correlates  to  the 
technical  perfonnance  measurement  (TPM).  It  has  been  argued  successfully  that  the 
quality  of  software  management  can  have  an  affect  on  the  degree  of  success  or  the 
possible  failure  of  a  software  development  program.  This  argument  was  presented  by 
Martin  J.  Machniak,  his  thesis  developed  metrics  for  measuring  the  quality  of  software 
management  along  four  dimensions:  requirements  management,  estimation/planning 
management,  people  management,  and  risk  management.  This  QMM  used  in  software 
development  for  program  managers  consists  of  a  composite  score  obtained  from  a 
questionnaire  administered  to  the  program  manager  and  their  peers.  The  QMM  reflects 
the  success  in  the  quality  of  software  management,  plus,  it  can  be  used  as  a  template  for 
possible  improvement  in  software  management  perfonnance.  The  author  administered  the 
same  questionnaire  survey  to  measure  the  conceptual  performance  of  the  individuals 
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responsible  for  Army  software  development  programs  on  the  government  side  of  the 
house.  The  author  also  identifies,  how  the  Technical  Perfonnance  Measures  (TPMs)  are 
applied,  and  how  (TPMs)  are  reported.  The  author  will  provide  data  how  this  process 
utilizes  TPMs:  (a)  as  key  measures  for  indicators  of  whether  or  not  a  program  is  a 
success  technically;  and  (b)  in  evaluating  a  program’s  ability  to  meet  requirements.  TPM 
metrics  are  used  to  track  and  compare  perfonnance  estimation,  predictions,  and  actual 
measurements  against  specified  and  allocated  goals  over  time.  The  author  feels  that  the 
conelation  between  QMM  and  TPM  can  provide  Program  Managers  (PM),  Integrated 
Product  Team  (IPT)  leaders,  and  customers,  with  good  objective  evidence  in  achieving 
design  quality  with  approved  requirements,  and  quality  program  management  using 
QMM  and  TPM  as  tools  for  program  success. 

E.  METHODOLOGY 

The  author  is  employed  at  the  U.S.  Army  Tank  Automotive,  Research  and 
Development  Engineering  Command  (TARDEC).  The  author  was  placed  on  a 
developmental  assignment  to  provide  software  quality  engineering  support  to  the 
developing  combat  systems  in  an  Integrated  Product  Teams  (IPT)  in  areas  of  Integration 
and  Simulation  Testing,  Modeling  and  Simulation,  and  Training. 

The  author  conducted  research  in  developing  this  thesis  from  various  Army 
programs  by  a  study  of  strategy  used  in  the  areas  of  Technical  Performance  Measurement 
(TPM),  and  administration  of  the  Quality  Management  Metrics  (QMM)  questionnaire 
surveys.  The  surveys  were  given  to  the  software  development  Program  Managers  (PM)  in 
software  acquisition,  to  determine  if  a  correlation  could  be  drawn  between  the  two 
metrics. 

The  major  challenge  encountered  and  overcome  during  the  completion  of  the 
thesis  was  the  consolidation  of  all  information  through  the  study  of  the  various  programs, 
plus  research,  and  arranging  interviews  with  very  busy  Program  Managers  working  under 
tremendous  pressure  to  do  it  right  the  first  time.  The  internet  provided  the  author  with 
good  reading  material  on  the  methodology  in  software  project  management  and  in 
software  project  management  strategy  in  general  for  industry  as  a  whole.  The  author 
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found  that  interviews  provided  insight  to  what  worked,  what  didn’t  work,  and  what  was 
too  costly  to  include  in  some  projects. 


F.  ORGANIZATION 

The  chapters  that  follow  describe  what  the  author  found  during  his  study  of  the 
various  programs,  which  included  administration  of  the  Quality  Management  Metrics 
(QMM)  and  Technical  Performance  Measurement  (TPM). 

CHAPTER  I:  Introduction:  problem,  solution,  research  questions,  scope, 
limitation,  and  assumptions,  methodology  and  organization  for  the  author’s  thesis. 

CHAPTER  II:  The  components  of  QMM,  TPM  and  data  from  both  metrics. 

CHAPTER  III:  Informal  Verification  and  Validation. 

CHAPTER  IV:  Conclusions,  Recommendations  and  Future  Work. 
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II.  METRIC  METHODOLGY  OF  QMM  AND  TPM 


A.  MOTIVATION 

The  author  felt  that  there  was  something  missing  in  the  software  program 
management  equation  that  might  have  been  over  looked  in  the  current  quest  for  cost- 
effective,  high-quality  software.  The  possible  missing  part  may  be  the  correlation 
between  QMM  and  TPM.  QMM  has  been  proven  by  Machniak  that  the  quality  of 
software  program  management  can  be  and  is  measurable,  and  available  for  input  in 
costing  and  scheduling  tools.  The  results  can  be  provided  to  program  managers  so  that 
they  may  pinpoint  such  areas  in  software  program  management  where  improvements  are 
needed,  and  can  be  made.  The  capability  to  measure  the  quality  of  management  of 
software  projects  objectively  allows  for  accurate  cost  models  where  impact  in 
management  quality,  including  cost  factors,  will  provide  a  means  for  software  project 
management  improvement  using  assessment  by  feedback  and  correction. 

Technical  Performance  Measures  can  be  used  to  develop  a  plan  of  expected 
technical  achievement  to  which  the  actual  progress  is  compared  using  periodic 
measurements  or  tests.  The  TPMs  are  indication  of  compliance  in  design  to  requirements 
captured  in  specifications  and  to  present  management  with  quantitative  data  to  determine 
whether  action  is  required.  The  TPM  approach,  using  various  techniques  of  risk  analysis 
and  probability,  offers  a  promising  method  that  incorporates  technical  assessments, 
resulting  systematically  from  technical  parameter  measurements  to  derive  more  discrete 
management  data  sufficiently  early  to  allow  for  cost  avoidance.  Therefore,  providing 
needed  infonnation  that  allows  the  managers  enough  time  for  making  informed  decisions 
on  schedule,  cost,  and  a  review  of  technical  requirements  early  in  the  program. 

In  this  thesis,  the  author  examines  the  possibility  of  a  correlation  between  TPM 
and  QMM  in  each  of  the  four  areas  covered  in  the  QMM  questionnaire  survey.  The 
author’s  intent  is  to  discover  if  any  of  the  four  areas  of  the  QMM  survey  have  a  stronger 
correlation  to  TPM  than  others,  in  identifying  contribution  to  management  quality  and 
possible  project  success. 
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B.  STRATEGY 

The  method  developed  in  approaching  the  correlation  between  QMM  and  TPM 
included  but  was  not  limited  too  reviewing  recommended  practices,  textbooks,  on-line 
publications,  and  having  interviews  with  various  personnel  from  senior  program 
managers  to  system  developers.  The  QMM  and  TPM  metrics  measured  the  quality  of 
management  plus  the  technical  perfonnance  on  three  specific  software  programs. 

The  author’s  goal  is  to  draw  an  objective  correlation  between  QMM  and  TPM  to 
which  program  management  can  be  compared  and  ranked  thus  giving  a  baseline  for 
quantifying  improvement.  In  the  next  few  paragraphs  an  explanation  will  be  given  for 
QMM  and  TPM  metrics. 

C.  QUALITY  MANAGEMENT  METRICS  (QMM) 

The  QMM  developed  by  Machniak  in  response  to  these  concerns  consists  of 
various  survey  questionnaires  covering  these  four  areas:  Requirements  management; 
estimation  and  planning  management;  people  management;  and  risk  management,  see 
Machniak  thesis  on  QMM  [REF  25]. 

The  QMM  survey  is  a  questionnaire  designed  to  be  given  to  software  project 
managers,  and  software  developers  who  have  global  impact  on  software  projects. 
Mackniak  applied  the  survey  on  three  software  programs  at  the  Space  and  Naval  Warfare 
Systems  Command  initially,  and  then  Grossman  validated  the  QMM  on  ten  software 
programs  at  Edwards  Air  Force  Base,  proving  furthermore  that  there  is  a  correlation  of 
good  quality  management  and  the  success  of  a  software  project. 

1.  Requirements  Management 

Software  requirements  management  focuses  on  managing  the  process  of 
extracting,  developing,  defining,  and  refining  the  requirements  of  a  software  program 
[REF  25],  It  is  the  current  belief  that  quality  management  of  a  program’s  requirements 
must  have  established  procedures  and  structure  to  ensure  that  requirements  specifications 
are  complete;  consistent;  understandable  to  the  reader;  lacking  ambiguity;  having  a 
known  origin;  and  not  having  vague  design  stipulations  [REF  25].  Also,  requirements 
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need  to  present  one  idea  per  requirement,  and  address  the  requirement  attributes.  Good 
requirement  management  provides  current  status  by  tracking  the  dates,  versions, 
relationships  to  other  requirements,  and  the  priority  rationale  behind  such  decisiveness. 

2.  Estimation/Planning  Management 

The  use  of  software  estimations  are  basically  one  of  the  main  methods  in  which 
planning  is  performed  in  software  programs.  The  QMM  estimation/planning  management 
section  will  not  give  a  specific  estimation  technique  as  being  the  right  one  over  others 
used.  However,  the  QMM  estimation/planning  management  section  will  seek  to  quantify 
management’s  efforts  in  the  estimation/planning  process.  In  other  words  the  questions  in 
the  survey  are  used  to  determine  if  the  choice  of  a  technique  is  appropriate  and  how  well 
that  technique  is  implemented  in  the  program. 

3.  People  Management 

In  QMM  quality  people  management  covers  the  need  for  organizational 
management  providing  a  good  atmosphere  with  proper  working  conditions  with  all 
environmental  efficiencies  maximized. 

In  QMM,  quality  people  management  has  the  work  flow  aided  by  delegation  and 
task  ownership  with  management  monitoring  those  activities  and  processes.  Questions 
QMM  ask:  Are  the  roles  well  defined  for  all  team  members?  Do  the  team  members’  have 
a  part  in  the  project  planning  and  decision  making  process?  Is  there  effective 
communication  being  given  from  top  down,  and  bottom-up  with  good  customer  or  team 
communication  occurring?  Also,  it  would  be  best  that  the  program  managers  have  a  good 
working  knowledge  in  the  technology  being  managed. 

4.  Risk  Management 

The  QMM  references  a  proactive  approach  on  quality  risk  management.  The  use 
of  a  formal  risk  management  plan  is  developed  usually  before  the  program  begins  with  a 
list  of  risks  identified  by  the  team  members,  and  customers  through  assessments  and  the 
use  of  checklists.  Throughout  the  life  of  the  development  risks  are  assessed  and  tracked 
by  management.  The  prioritization  of  risks  is  based  on  the  probability  of  occurrence  and 
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negative  consequences.  A  risk  strategy  is  formulated  to  mitigate  risks  with  a  plan 
developed  to  allocate  needed  resources  in  reflection  of  risk  priorities.  Risk  data  is  to  be 
shared  [REF  25]. 

The  author  notes  that  the  QMM  was  developed  to  reflect  the  management  needs 
of  large  projects  and  tests  through  the  questionnaire  survey,  and  formal  methods  are  used 
for  the  management  of  requirements,  performing  estimations  and  planning,  managing 
people,  and  managing  risk. 

D.  QMM  SURVEYS 

Software  program  managers  on  software  development  programs  at  U.S.  Army 
TACOM  were  asked  to  complete  the  QMM  survey.  These  individuals  were  selected 
because  of  their  complete  understanding  of  the  program  and  the  fact  that  they  had  a  good 
understanding  of  the  management  practices  which  were  implemented  throughout  the 
software  program.  The  software  program  managers  used  a  specific  point  in  time  in  the 
program  for  the  evaluation  of  the  program  management,  so  that  the  individual  team 
members  were  able  to  identify  the  selected  point  in  time  and  evaluate  the  program.  Also, 
the  TPM  evaluation  was  selected  during  the  same  point  in  time. 

In  the  best  interest  of  the  program  and  to  maintain  confidentiality  of  the  survey 
the  programs  are  identified  as  programs  A,  B,  and  C. 

The  interviewees,  after  completing  the  survey,  were  asked  to  rate  the  success  of 
that  period  or  selected  point  in  time  using  an  evaluation  scale  of  zero  to  ten.  The  score  of 
zero  corresponded  to  program  failure  and  ten  corresponded  to  a  completely  successful 
program.  A  score  of  ten  meant  that  the  software  program  produced  a  product  on  time, 
and  within  the  budget  allocated  as  well  as  complete  customer  satisfaction  with  the  quality 
product. 

Part  I  of  the  QMM  questionnaire  survey: 

This  part  of  the  survey  questionnaire  is  the  pair-choice  questions.  It  consists  of 
two  questions  placed  side  by  side  on  a  single  line  within  a  column  next  to  each  question. 
The  interviewees  were  asked  to  check  a  box  next  to  the  question  or  statement  which 

10 


closely  reflected  what  was  happening  on  the  evaluation  program  at  the  specific  point  in 
time.  The  interviewees  made  choices  of  the  two  for  each  line  of  the  survey  questionnaire. 
The  question  or  answer  that  most  likely  reflected  the  evaluation  program  need  not  be  an 
exact  match.  There  were  two  different  ideas  for  each  pair-choice  question.  This  was 
done  in  an  effort  to  find  a  tendency  of  the  interviewee  in  the  area  of  interest  by  way  of 
formal  requirements  documentation  versus  infonnal  requirements  or  documentation. 
Most  often  the  pair-choice  questions  were  repeated  with  different  wording  to  confirm  the 
earlier  choices  and  measure  the  strongest  tendency.  The  format  of  the  questionnaire 
survey  using  the  proper  mix  of  questions,  plus  a  variation  with  repetitions,  was  designed 
to  reach  a  consensus  on  issues,  measure  tendencies,  and  show  strengths  [REF  25]. 

Part  II  of  QMM  Questionnaire  Survey: 

This  part  of  the  survey  questionnaire  is  basically  yes  or  no  questions  that  consist 
of  one  question  per  line  with  three  columns  next  to  it  giving  the  person  a  possible  “yes,” 
“no,”  and  “N/A”  answer  [REF  25]. 

The  interviewee  answering  the  questions,  would  answer  as  it  pertains  to  a 
program  manager  and  the  program  during  a  specific  point  in  time  on  the  program  by  a 
“yes,”  “no,”  and  “N/A”  in  the  box  next  to  each  question,  with  the  use  of  the  “N/A”  box 
discouraged  unless  the  program  manager  has  no  say  in  such  issues. 

In  the  requirements  management  pair-choice  section,  a  score  of  zero  to  two  is 
possible  having  different  upper  bounds  on  the  score  of  each  question.  This  is  based  upon 
the  relative  weight  and  importance  of  each  question  in  the  section.  However,  in  the 
estimation/planning  management,  people  management  and  risk  management  sections  the 
possible  scores  were  zero  to  one. 

The  questions  that  answer  either  yes  or  no  have  a  score  that  ranges  from  minus 
four  to  plus  four.  This  score  is  based  upon  its  relative  weight  and  importance  in  the  upper 
and  lower  bounds  of  the  survey  questionnaire. 

This  was  detennined  by  Machniak  [REF  25]  and  stated  as  such  [Q,M,M&G].  The 
QMM  equation  is  given  by: 
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QMM=0.92RQM+0.67EPM+0.55RKM+ 1.86PM,  where: 
RGM  is  the  requirements  management  metric, 
EPM  is  the  estimation/planning  metric 


RKM  is  the  risk  management  metric; 

PM  is  the  people  management  metric 

Having  coefficients  ranging  from  0.92,  0.67,  0.55  and  1.86  as  the  importance 
coefficients  of  the  requirements,  estimation/planning,  risk  and  people  management 
metrics  respectively.  As  the  importance  coefficients  have  been  detennined  through  focus 
groups,  interviews  with  one-on-one  experienced  software  professionals  [REF  25]. 


Data  Analysis 


Program 

Program  A 

Program  B 

Program  C 

Participant 

ApM 

Aj 

Bpm 

Bi 

CpM 

Ci 

QMM  score 

509.65 

522.63 

569.03 

559.44 

314.83 

229.21 

QMM  percent 

77.35 

79.32 

86.36 

84.91 

47.78 

45.36 

Success  score 

8 

8 

9 

8 

6 

6 

Mean  success  score 
(0-10) 

8 

8.5 

6 

Table  1.  Results  of  Informal  QMM  Validation 


Table  1  is  the  summary  of  the  three  programs  included  in  this  analysis  using  data 
from  the  program  manager  and  independent  development  team  members.  In  all  of  the 
following  tables  QMM  percentage  score,  requirements  management,  estimation/planning 
management,  people  management,  and  risk  management  scores  have  all  been  adjusted  to 
a  scale  of  zero  to  ten.  The  zero  score  in  Table  1  corresponds  to  zero  percent  of  the  points 
found  possible  in  the  section  where  as  a  score  of  ten  corresponds  to  a  possible  100 
percent  or  100  points  in  the  section. 


Program 

PM 

Program 

Score 

PM 

QMM 

Score 

PM 

Requirements 

Management 

PM 

Estimation 

Planning 

PM 

People 

Management 

PM 

Risk 

Management 

A 

8 

7.7 

88 

66 

169 

56 

B 

9 

8.6 

101 

80 

193 

64 

C 

6 

4.8 

49 

70 

6 

60 

Table  2.  Program  Manager  Summary  Data 
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Table  2  is  a  summary  of  the  Program  Manager  Data.  The  first  column  from  left 
to  right  identifies  the  program,  the  second  column  provides  the  program  managers 
subjective  program  score,  the  third  column  provides  the  QMM  score  based  on  the 
program  managers  questionnaire  survey,  while  columns  four  through  seven  provides  the 
program  managers  QMM  requirements  management,  estimation  and  planning 
management,  people  management  and  risk  management  scores  reflecting  the 
questionnaire  survey. 

E.  TECHNICAL  PERFORMANCE  MEASUREMENT  (TPM) 

Technical  Performance  Measurement  (TPM)  is,  in  its  most  basic  fonn  a  plan  of 
expected  technical  achievement  to  which  the  actual  progress  is  compared  using  periodic 
measurements  or  tests,  see  [REF  20]. 

Technical  performance  measures  are  engineering  and  physical  measures,  such  as 
computer  throughput,  radar  detection  range,  number  of  possible  users  and  programmatic 
metrics  used  by  a  program  in  gauging  effectiveness  in  developing  designs  to  ensure  that  a 
design  meets  the  performance  specified  by  the  customer.  The  TPMs  are  indicative  of 
compliance  in  design  to  requirements  captured  in  specifications  and  presents 
management  with  quantitative  data  to  determine  whether  action  is  required.  The  TPM  has 
been  integrated  with  requirements  management  issue  and  action  management,  baseline 
management,  and  risk  management.  TPMs  evaluate  the  adequacy  in  evolving  solution 
through  engineering  changes  and  trade  studies  to  identify  deficiencies  that  impact  the 
systems  ability  to  meet  the  performance  requirement.  Technical  characteristics  are 
evaluated  to  identify  problems  through  engineering  analyses  and  should  indicate  if 
performance  is  being  met  as  specified  in  contracts  or  other  requirements.  As  the  system 
concept  is  being  developed  the  TPMs  are  initially  defined,  and  are  formalized  during 
contract  start  through  requirements  definition.  Existing  TPMs  can  be  modified  per 
program  needs,  and  new  TPMs  can  be  added  to  the  system  at  any  time  during  the 
program. 

Technical  Performance  Measurement  (TPM)  supports  the  Army’s  objective  and 
strategy  by  establishing  adaptive  and  affordable  processes.  It  also  supports  the  monitor 
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and  control  process  area  in  the  system  engineering  process.  In  the  government  systems 
program  managers  and  their  teams,  find  themselves  in  an  environment  that  creates 
pressures  that  can  be  translated  into  products  being  delivered  using  best  value  analysis 
with  cost  as  the  overriding  determinant.  (The  TMP  approach,  using  various  techniques  of 
risk  analysis  and  probability,  offers  a  promising  method  that  incorporates  technical 
assessments,  resulting  systematically  from  technical  parameter  measurements  to  derive 
more  discrete  management  data  sufficiently  early  to  allow  for  cost  avoidance.)  Therefore, 
this,  in  essence,  provides  needed  information  that  allows  the  managers  more  time  for 
making  informed  trade-off  decisions  early  in  the  program. 

A  few  recent  initiatives  are  breaking  new  ground  in  the  development  of 
sophisticated  techniques  for  TPM  planning,  integration  with  cost  and  schedule,  in  such 
manner  to  be  reflected  in  Earned  Value  Management  (EVM)  data. 


Earned  Value  Management: 

a.  Earned  Value  Management  (EVM)  is  an  integrated  system  of  project 
management  and  control  which  has  enabled  the  contractor  and  their  customer 
to  monitor  the  progress  of  a  project  in  terms  of  integrated  cost,  schedule  and 
technical  performance  measures,  see  Appendix  B. 


Integrated  Product  Team  (IPT)  Ownership: 

b.  EVM  system  is  created,  owned  and  managed  by  the  Prime  Contractor,  but  the 
customer  has  full  and  timely  visibility  of  the  information  at  any  time.  From 
this  perspective  this  means  that  there  is  greater  equality  of  information 
between  the  contractor  and  the  customer  which  is  fundamental  to  true 
partnering. 

EVM  function: 

c.  EVM  provides  a  reference  point  which  is  an  objective  view  of  the  status  of  the 

contract  such  that  the  value  to  the  end  or  goal  reflects  work  completed  to  date. 

This  needs  to  be  compared  with  both  the  planned  expenditure  and  the  actual 
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costs  to  determine  the  performance  to  date  and  to  give  early  indications  of 
problems.  Now  EVM  may  also  be  used  to  enhance  cost  forecasting,  risk 
management  and  as  the  basis  for  payment  against  the  contract. 


EVM  data  requirement: 

d.  The  way  in  which  EVM  is  implemented,  the  contractor  must  have  a  validated 
system  that  can  accurately  measure  the  following  three  fundamental  factors: 

1 .  The  Budgeted  Cost  of  Work  (BCWS)  or  what  is  known  as  planned  costs. 

2.  The  Actual  Cost  of  Work  Performed  (ACWP)  or  what  is  known  as  the 
actual  cost  of  progress  made. 

3.  The  Budgeted  Cost  of  Work  Performed  (BCWP)  or  earned  value. 


Earned  Value  Management  system: 

e.  The  heart  of  EVM  is  the  Work  Breakdown  Structure  (WBS).  The  WBS  is  a 
product  oriented  family  tree  structure  of  all  of  the  goods  and  services  to  be 
built  or  supplied.  The  WBS  is  a  consistent  and  visible  framework  that 
displays  and  defines  the  products  as  elements  that  relate  to  the  end  product. 

f.  The  WBS  needs  to  be  defined  down  to  at  least  the  level  at  which  EVM 
reporting  will  be  applied,  and  within  a  WBS  that  adequately  meets  their  data 
requirements. 

g.  The  schedules  that  are  produced  for  the  lower  elements  of  the  WBS  should  be 
planned  to  the  greatest  possible  detail  in  that  the  resulting  activities  are  of 
manageable  duration  and  can  be  assigned  to  a  single  part  of  the  organization. 

h.  Earned  value  is  based  on  assigning  a  value  at  the  activity  level  to  the 
achievement  of  project  work.  Ideally,  to  determine  non- subjective 
achievement  such  as  milestones  and  deliverables,  one  would  need  to  have 
them  based  on  the  planned  cost  (in  money  or  hours)  for  achieving  the  goal. 

i.  The  control  sometimes  called  Cost  Account  (CA)  coincides  with  the  level  at 
which  EVM  reporting  will  be  applied.  The  CA  has  a  dedicated  manager 
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appointed.  This  individual  is  empowered  to  plan  and  deliver  within  time  and 
cost,  those  constraints  set  forth  within  the  CA. 

EVM  is  generated: 

j.  This  begins  once  the  project  is  underway  -the  contractor  will  start  to  earn 
value  by  the  commencement  and  completion  of  individual  activities.  The 
summation  of  the  values  earned  in  a  particular  control  account  gives  the 
earned  value  of  the  CA  to  date. 


EVM  data  is  presented: 

k.  The  earned  value  is  plotted  against  the  planned  and  actual  costs  over  time. 

This  is  a  very  clear  way  to  show  the  status  of  the  project.  The  progress  report 
has  a  basic  tabulation  of  the  three  basic  data  elements,  the  estimate  at 
completion  and  the  budget,  and  their  derived  data  elements,  or  variances, 
which  are  measured  in  terms  of  resources  such  as  man-hours  or  cost.  The 
derived  data  elements  are: 

1 .  Cost  Variance  (CV)  -  The  difference  between  the  planned  and  actual 
resource  usage  for  an  element  of  work.  A  negative  variance  means  that 
more  money  was  spent  for  the  work  accomplished  than  was  planned.  Cost 
Variance  is  obtained  by  comparing  actual  cost  with  earned  value: 

2.  Cost  Variance  =  Earned  Value  -  Actual  Cost 

3.  CV  =  BCWP  -  ACWP 

4.  Schedule  Variance  (SV)  -  The  difference  between  the  budget  and  the 
earned  value  for  an  element  of  work  is  called  the  schedule  variance. 

5.  Schedule  Variance  =  Earned  Value  -  Budget 

6.  SV  =  BCWP  -  BCWS 

7.  Variance  at  Completion  (VAC)  -  The  difference  between  the  total  budget 
allocated  for  a  piece  of  work  and  the  project  manager’s  estimate  of  the 
actual  resource  cost  at  completion. 
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An  example  of  the  way  EVM  data  may  be  presented  in  the  fonn  of  a  graph  is 
shown  in  Figure  1.  This  can  be  available  at  the  total  contract  level  and  at  all  WBS  levels 
down  to  the  lowest  level  set  within  the  contract. 


Time  Planned 

‘now'  completion 


KEY 

EAC  Estimate  at  Completion 
BAC  Budget  at  Completion  (current) 

BCWS  Budgeted  Cost  of  Work  Scheduled  (current) 

BCWP  Budgeted  Cost  of  Work  Performed  (earned  value) 

ACWP  Actual  Cost  of  Work  Performed 

Figure  1.  EVM  Graphical  Representation 
EVM  data  in  graphical  representation: 

l.  These  graphical  representations  are  useful  management  information  tools.  For 
example,  in  Figure  1,  the  graph  may  represent  a  project  or  task  that  appears  to 
be  underachieving  in  terms  of  both  cost  and  schedule.  Now  if  corrective 
action  is  not  taken,  the  project/task  will  be  completed  behind  schedule  and 
over  budget. 

m.  As  well  as  the  derived  performance  indicators  mentioned  above,  there  are  two 
measures  of  efficiency  which  are  also  useful  for  determining  the  status  of  the 
project:  (a  ratio  of  less  than  one  implies  that  work  is  underachieving  against 
the  plan,  and  above  one  implies  better  than  the  plan). 

1 .  Cost  Performance  Index  (CPI)  =  How  much  it  really  costs  to  earn  one 
pound  of  budget  or  the  “value  for  Money”  report. 

2.  Cost  Performance  Index  =  Earned  Value  /  Actual  Cost 

3.  CPI  =  BCWP  /  ACWP 
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4.  Schedule  Performance  Index  (SPI)  -  The  Schedule  Performance  Index  is 
the  ratio  of  Earned  Value  and  the  Planned  Achievement. 

5.  Schedule  Performance  Index  =  Earned  Value  /  Budget 

6.  SPI  =  BCWP  /  BCWS 

Reporting  Cycle: 

n.  The  reporting  cycle  should  as  a  minimum  tie  in  the  contractors  and  customers 
internal  accounting  periods  (usually  monthly  -  although  on  the  high  risk 
projects  the  reporting  cycle  is  weekly). 

Cost  &  Schedule  Perfonnance  Index  Chart: 

o.  The  CPI/SPI  trend  chart  in  (Table  3)  provides  a  summary  of  the  three 
programs  A,  B  and  C  that  are  included  in  this  analysis  using  data  from  TPM 
presented  though  EVMS: 

1 .  SPI  =  (BCWP/BCWS)  and  CPI  =  BCWP/ACWP. 

2.  The  (BCWS)  Budgeted  Cost  for  Work  Scheduled,  which  is  distributed 
cost  for  all  the  work  that  is  realistically  time-phased  based  on  schedule, 
scope  and  resources. 

3.  The  (BCWP)  Budgeted  Cost  for  Work  Performed;  also  referred  to  as 
Earned  Value.  The  budgeted  cost  for  all  the  work  actually  accomplished  in 
a  given  period  of  time,  as  a  measurement  of  work  progress. 

4.  The  (ACWP)  Actual  Cost  of  Work  Perfonned;  also  referred  to  as  (ACI) 
Actual  Cost  Incurred  and  can  mean  actual  cost  as  recorded  in  the 
accounting  system  for  all  the  work  associated  with  a  given  period  of  time. 
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Prog 

Mth. 

OCT 

NOV 

DEC 

JAN 

FEB 

MAR 

APR 

MAY 

JUN 

JUL 

AVG 

COE 

A 

SPI 

98% 

100 

% 

100 

% 

121 

% 

117.5 

% 

114.5 

% 

100% 

100% 

100% 

100% 

105 

1.05 

CPI 

108.5 

% 

107 

% 

101.5 

% 

123.5 

% 

113.5 

% 

114% 

113.5 

% 

113.5 

% 

113.5 

% 

113.5 

% 

112.5 

1.13 

B 

SPI 

100 

% 

100 

% 

100 

% 

100 

% 

100% 

100% 

100% 

100% 

100% 

100% 

100 

1.00 

CPI 

100 

% 

100 

% 

100 

% 

100 

% 

100% 

100% 

100% 

100% 

100% 

99.9 

% 

99.99 

1.00 

C 

SPI 

98.5 

% 

98.2 

% 

98.2 

% 

98.2 

% 

97.1 

% 

97.1 

% 

96% 

97.5 

% 

97.5 

% 

97.5 

% 

97.37 

.974 

CPI 

95  % 

96.2 

% 

96.1 

% 

97.1 

% 

101% 

100.5 

% 

104% 

104.5 

% 

105% 

104.5 

% 

100.2 

1.00 

Table  3 .  SPI/CPI  Trend  Chart 


In  Table  3,  the  author  took  the  average  of  each  program’s  SPI  and  CPI  over  the 
ten  month  period,  and  then  the  coefficient. 

F.  SUMMARY: 

In  section  II  the  author  presented  the  metrics  that  are  to  be  used  to  answer  the 
question...  “Is  there  a  correlation  between  Quality  Management  Metrics  (QMM)  and 
Technical  Performance  Measurement  (TPM).” 

1 .  The  quality  of  software  management  has  an  effect  on  the  degree  of  success  or 
failure  of  a  software  development  program,  this  statement  has  been  argued 
successfully  by  Martin  J.  Machniak  in  his  thesis  Development  of  QMM 
Measuring  Software  Program  Management  Quality.  The  QMM  metrics  can 
be  used  both  to  characterize  the  quality  of  software  management  and  provide  a 
template  for  improving  software  management  performance  [REF  25]. 

2.  TPM  uses  engineering  data  that  physically  measures:  computer  throughput, 
radar  detection  range,  number  of  users  and  other  programmatic  metrics  such 
as  EVMS.  This  helps  the  program  manager  gauge  the  effectiveness  of  a 
developing  design  in  meeting  the  performance  specifications  developed  for 
the  U.S.  Anny.  The  TPM  is  used  as  an  indicator  for  compliance  of  a  design  to 
requirements  or  specifications  and  presents  management  with  quantitative 
data  that  can  be  used  to  determine  if  corrective  action  is  needed.  TPM  is 
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integrated  with  EVMS  reflecting  cost  and  scheduling,  design  requirements, 
issue  and  action  management,  baseline  management,  and  risk  management  for 
impact  assessment  [REF  20]. 

The  author  administered  the  QMM  questionnaire  to  measure  the  perceptions  of 
program  managers  from  programs  A,  B,  and  C  that  have  the  responsibility  for  the 
software  development  within  each  of  the  said  programs  for  the  U.S.  Army.  The  author 
then  gathered  TPM  data  using  a  metric  methodology  from  the  same  programs  given  the 
questionnaire,  and  developed  the  data  tables  for  possible  correlation  between  them  if  any. 

In  Section  III  the  author  presents  the  informal  verification  and  validation. 
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III.  INFORMAL  VERIFICATION  AND  VALIDATION 


A.  MOTIVATION 

The  methodology  and  structure  for  evaluating  the  possible  correlations  between 
Technical  Performance  Measurement  (TPM)  and  Quality  Management  Metrics  (QMM) 
have  been  laid  out  in  the  previous  chapter,  with  the  informal  verification  and  validation 
presented  in  this  section. 

Informal  verification  and  validation  being  necessary  ensuring  that  both  metrics 
TPM/EVM  and  QMM  reflect  a  positive  correlation  between  each  other  having  measured 
the  cost  and  scheduling  with  TPM/EVM  and  the  quality  of  software  program 
management  in  a  fashion  as  accurately  as  possible  using  QMM. 

B.  STRATEGY 

The  verification  and  validation  approach  is  informal.  The  evaluation  was  of  three 
software  programs  using  the  QMM  survey  score  and  the  TPM/EVM  metrics  from  the 
same  three  programs  during  the  same  time  period.  The  program  manager  and  one 
program  developer  from  the  same  team  evaluated  program  A,  and  such  was  the  case  for 
programs  B  and  C,  using  the  program  manager  and  one  program  developer. 

In  developing  a  frame  of  reference  for  which  a  correlation  can  be  established  from 
the  QMM  survey  results,  two  measures  were  used.  The  two  measurements  are  the  1) 
QMM  percentage  score,  and  2)  the  overall  program  success  score. 

1.  The  QMM  percentage  score  is  derived  by  first  taking  the  surveys  minimum 
QMM  score  and  normalizing  it  to  zero.  This  can  be  done  by  adding  130.86  to 
the  minimum  score  of  -130.86  in  doing  so  makes  it  zero.  The  maximum 
QMM  score  possible  from  the  survey  is  528.00,  adding  130.86  for 
nonnalization,  the  survey  maximum  possible  score  is  now  658.86. 

2.  The  QMM  percentage  score  is  obtained  by  dividing  the  minimum  normalized 
score  by  the  maximum  nonnalized  score,  and  multiplying  the  results  by  a 
hundred. 

The  equations  taken  from  Martin  J.  Machniak  thesis: 
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QMM  (min)  +  130.86  =  0.00  =  QMM  (min  normalized) 

QMM  (max)  +  130.86  =  658.86  =  QMM  (max  normalized) 

QMM  (score)  +  130.86  =  QMM  (score  normalized) 

(QMM  score  normalized  /  QMM  max  nonnalized)  X  100  =  QMM  %  score. 

The  participant  taking  the  survey  assigns  an  overall  program  success  score  based 
on  how  they  feel  the  program  is  doing  from  their  perspective  and  is  totally  subjective. 
However,  for  the  most  part  the  success  of  a  program  is  measured  by  the  final  product 
performance  and  whether  or  not  it  meets  the  user’s  satisfaction  and  the  stakeholder’s 
expectation. 

A  comparison  is  made  between  the  QMM  survey  score  and  the  individual  overall 
success  score,  and  to  the  mean  overall  success  score  of  the  program. 

The  mean  overall  success  score  of  the  program  is  based  on  surveys  from  the 
project  manager  and  other  individuals  capable  of  judging  the  overall  success  of  the 
program.  The  scale  used  to  measure  the  overall  success  of  a  program  by  the  individual 
taking  the  survey  is  presented  by  a  score  from  zero  to  ten.  The  best  program  would  be 
given  a  score  of  ten,  with  a  zero  score  being  a  program  failure.  However,  the  author 
would  like  to  make  it  clear  that  an  overall  success  score  of  ten  is  defined  as  having 
perfect  software  product  and  program  execution,  and  that  success  or  failure  of  a  software 
program  is  not  always  due  to  the  actions  of  program  management. 

The  comparison  of  the  three,  QMM  percentage  score,  the  individual  score,  and 
the  mean  overall  success  score  of  the  program  will  establish  any  correlation  between 
them  for  each  program.  The  example,  given  by  Martin  J.  Machniak  in  his  thesis,  dated 
December  1999,  stated  that  the  possible  overall  success  score  of  seven  corresponded  to  a 
QMM  percentage  score  of  70  percent  plus  or  minus  5  percent  would  indicate  a  strong 
correlation.  An  overall  success  of  seven  to  QMM  percentage  greater  than  a  plus  or  minus 
five  percentage  points  of  70  percent,  and  less  than  plus  or  minus  15  percentage  points  of 
70  percent  can  be  considered  a  fair  correlation.  However,  in  a  program  where  8  is  the 
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overall  success  score,  with  relationship  to  a  QMM  percentage  score  of  40 
percent,  the  correlation  is  considered  weak. 

The  Technical  Performance  Measurement  (TPM)  metrics  was  evaluated  based  on 
the  Earned  Value  Management  (EVM)  data  which  is  an  integrated  system  to  monitor  the 
progress  of  a  project  in  tenns  of  integrated  cost,  schedule  and  technical  performance 
measures.  The  author  would  like  to  note  that  traditional  project  management  practices 
tend  to  compare  the  actual  costs  with  planned  expenditures,  and  sometimes  confuses 
actual  known  costs  with  actual  known  progress.  In  as  much  as  actual  costs  are  not 
necessarily  in  some  cases  good  measures  of  progress,  the  EVM  can  provide  a  third 
reference  point  which  is  an  objective  view  of  the  project  status;  an  example  would  be  the 
value  to  the  end  goal  of  the  work  completed  to  date.  Using  EVM,  problems  can  be 
indicated  early  by  comparing  both  the  planned  expenditure  and  the  actual  costs  to 
determine  the  performance  to  date  of  the  project. 

The  project  manger,  in  order  to  implement  EVM,  needs  to  have  a  validated 
system  that  accurately  measures  the:  1)  planned  cost  of  work,  also  known  as  the 
Budgeted  Cost  of  Work  Scheduled  (BCWS);  2)  the  actual  cost  of  the  progress  made,  also 
known  as  the  Actual  Cost  of  Work  Performed  (ACWP);  3)  The  earned  value,  also  known 
as  the  Budgeted  Cost  of  Work  Performed. 

The  author  states  that  the  Work  Breakdown  Structure  (WBS)  provides  a  sort  of 
family  tree  where  all  the  goods  and  services  are  to  be  supplied.  This  family  tree  gives  a 
visible  framework  to  display,  and  define  the  products  and  elements  that  make  up  the  end 
product.  Ideally  Earned  Value  is  assigned  value  at  an  activity  level  to  an  achievement  of 
project  work;  and  is  non-subjective;  based  on  milestones;  deliverables;  and  based  on 
planned  costs,  such  as  money  or  hours  of  achieving  that  milestone  or  deliverable.  The 
Earned  Value  techniques  are  numerous  and  can  be  applied  to  various  activities  with  the 
guidance  from  a  specialist  in  that  particular  activity. 

The  author  was  given  TPM/EVM  data  that  was  available  from  Control 
(sometimes  called  Cost)  Account  (CA)  in  programs  A,  B  &  C.  The  programs  appointed 
managers  from  each  program  A,  B  and  C  provided  two  indicators: 
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1 .  Cost  Performance  Index  (CPI)  which  is  how  much  it  really  costs  to  earn  one 
pound  of  budget  or  the  “Value  for  Money”  report:  Cost  Performance  Index  = 
Earned  Value/Actual  Cost,  CPI  =  BCWP/ACWP. 

2.  Schedule  Performance  Index  (SPI)  that  shows  the  schedule  Perfonnance 
Index  as  the  ratio  of  Earned  value  and  the  Planned  Achievement:  Schedule 
Perfonnance  Index  =  Earned  Value/Budge,  SPI  =  BCWP/BCWS. 

C.  RESULTS 

In  the  following  paragraphs  are  the  results  of  the  QMM  surveys  and  the 
TPM/E  VMS  data. 

1)  The  scores  form  the  QMM  survey  presented  in  Table  4  summary  for  the  A,  B, 
and  C  programs.  The  QMM  was  detennined  for  each  of  the  three  programs  A,  B,  and  C 
using  QMM  score  as  a  percentage  of  the  QMM  maximum  possible  score  of  each 
program.  The  percentages  of  each  program  were  compared  to  the  scores  given  by  survey 
participants  for  a  comparison.  This  provides  a  mean  success  score  for  each  of  the 
programs  too  include  both  the  Project  managers  and  other  associates  within  each  program 
that  have  the  insight  forjudging  program  success. 


Program 

Program  A 

Program  B 

Program  C 

Participant 

ApM 

Ai 

Bpm 

Bi 

CpM 

Ci 

QMM  score 

509.65 

522.63 

569.03 

559.44 

314.83 

229.21 

QMM  percent 

77.35 

79.32 

86.36 

84.91 

47.78 

45.36 

Success  score 

8 

8 

9 

8 

6 

6 

Mean  success 
score  (0-10) 

8 

8.5 

6 

Table  4. 


QMM  Results  Summary  Comparison. 


The  author  notes  that  the  survey  results  for  all  programs  reflect  a  correlation 
between  the  QMM  percentage  ranking,  the  overall  success  ranking  of  the  program, 
individual  success  ranking  scores,  and  the  mean  ranking  scores. 

All  QMM  survey  summary  sheets  for  programs  A,  B,  and  C  are  enclosed  and 
presented  in  Appendix  C. 
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a.  The  examination  of  the  survey  summary  sheets  for  program  A,  found  that 
there  was  a  slightly  lower  score  in  the  areas  of  people  and  requirements 
management.  But,  the  end  product  was  good  due  mostly  to  experienced 
personnel  with  a  history  of  working  together  both  as  stakeholders,  users  and 
technical  staff. 

b.  The  examination  of  program  B  reflected  very  good  scores  in  all  four  areas  of 
the  QMM  survey,  and  provided  an  excellent  product  with  a  timely  delivery. 
Once  again  the  program  had  experienced  personnel  in  all  areas  of 
requirements  management,  people  management,  risk  management, 
estimation/planning  management,  and  supported  by  an  excellent  technical 
staff.  However,  the  author  would  like  to  point  out  that  a  program  where 
people  are  this  experienced  may  have  the  attitude  that  they  have  seen  it  all 
before  and  the  Project  Manager  needs  to  have  very  strong  leadership  skills 
with  a  reputation  of  known  success  in  order  to  guide  them. 

c.  Program  C  scored  poorly  in  two  areas  of  the  QMM  survey,  and  this  was 
reflected  in  the  QMM  score,  QMM  percentage  score,  the  success  score  given 
by  the  participants,  and  the  mean  success  score.  The  first  was  requirements 
management,  and  second  was  people  management.  In  requirements 
management  the  problem  issues  stem  from  not  having  very  well  defined 
technical  goals  and  constant  changing  program  requirements.  The  changes 
made  in  the  technical  goals  without  communicating  with  all  the  stakeholders 
and  the  users  left  the  technical  part  of  the  program  unable  to  establish  TPM’s 
or  EVM’s.  The  lack  of  well  defined  requirements  and  immature  technology 
caused  personnel  to  request  a  transfer  from  the  program.  The  level  of 
frustration  in  all  areas  of  the  program  made  the  turnover  of  personnel  very 
common  and  to  the  point  where  training  was  done  by  new  hires  for  other  new 
hires. 

The  author  found  the  self  enhancement  bias  stated  in  Martin  J.  Machniak  thesis  to 
be  true.  In  all  interviews  with  program  managers  most  felt  that  they  could  always  solve 
the  other  program  managers  program  problems.  But,  when  asked  about  their  own 
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program  problems  stated  simply  theirs  were  unique  only  to  their  program.  The  need  for  a 
QMM  survey  administrator  to  explain  the  intent  is  a  must,  and,  interviewing  the  people 
before  and  after  the  QMM  survey  is  necessary  in  order  to  have  everyone  become  aware 
of  the  differences  as  perceived  in  what  is  thought  to  be  happening  in  a  program  and  what 
is  actually  taking  place  and  what  is  required  in  the  program.  The  after  actions  review  with 
all  participants  of  the  survey  discussing  questions  and  answers,  for  the  most  part  was  the 
biggest  benefit  of  the  QMM  process.  All  QMM  summary  sheets  for  programs  A,  B,  and 
C  for  all  survey  participants  are  found  in  Appendix  C. 

All  copies  of  the  completed  survey  from  each  of  the  three  program  managers  and 
other  program  participants  are  included  in  Appendix  A.  Also,  QMM  survey  questionnaire 
templates  with  points  and  ranking  of  each  response  can  be  found  in  Appendix  A. 

2)  The  data  from  the  TPM/EVM  is  presented  in  Table  5  summary  for  A,  B,  and  C 
programs.  The  CPI  and  the  SPI  average  plus,  coefficient  of  each  program  over  a  ten 
month  period  included  in  this  analysis  using  data  from  TPM  presented  though  EVMS.  All 
EVM  summary  sheets  for  programs  A,  B,  and  C  are  enclosed  and  presented  in  Appendix 
C. 

The  EVM  perfonnance  goal  was  determined  for  each  of  the  three  programs  A,  B, 
and  C  using  a  ratio  of  less  than  one  implies  that  work  is  underachieving  against  the  plan, 
and  above  one  implies  better  than  the  plan. 


Prog 

Mth. 

OCT 

NOV 

DEC 

JAN 

FEB 

MAR 

APR 

MAY 

JUN 

JUL 

AVG 

COE 

A 

SPI 

98% 

100% 

100% 

121% 

117.5 

% 

114.5 

% 

100% 

100% 

100% 

100% 

105 

1.05 

CPI 

108.5 

% 

107% 

101.5 

% 

123.5 

% 

113.5 

% 

114% 

113.5 

% 

113.5 

% 

113.5 

% 

113.5 

% 

112.5 

1.13 

B 

SPI 

100% 

100% 

100% 

100% 

100% 

100% 

100% 

100% 

100% 

100% 

100 

1.00 

CPI 

100% 

100% 

100% 

100% 

100% 

100% 

100% 

100% 

100% 

99.9 

% 

99.99 

1.00 

C 

SPI 

98.5 

% 

98.2 

% 

98.2 

% 

98.2 

% 

97.1 

% 

97.1 

% 

96% 

97.5 

% 

97.5 

% 

97.5 

% 

97.37 

.974 

CPI 

95% 

96.2 

% 

96.1 

% 

97.1 

% 

101% 

100.5 

% 

104% 

104.5 

% 

105% 

104.5 

% 

100.2 

1.00 

Table  5. 
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The  author  reviewed  the  data  from  Table  2,  which  provided  a  summary  of  the  CPI 
and  the  SPI  average  plus,  coefficient  of  each  program.  Earned  value  is  a  means  of  placing 
a  dollar  on  project  status,  in  this  way  provides  project  manager’s  a  way  to  compare 
budget  versus  actual  costs  versus  what  the  project  status  is  in  dollar  amounts.  In  order  to 
have  a  proper  analysis  of  the  project  the  following  items  will  be  needed:  budget,  earned 
value,  actual  costs,  and  forecasts.  A  reference  to  Figure  2  indicates  it  is  without  earned 
value,  it  shows  actual  costs  as  less  than  what  has  been  budgeted,  and  it  is  impossible  to 
tell  if  the  actual  costs  are  less  or  if  work  is  progressing  at  a  slower  rate  than  planned  or 
actual  costs  are  less  than  what  was  budgeted.  Earned  value  can  be  defined  as  the  sum  of 
the  budgets  for  the  work  that  is  complete,  and  earned  value  for  completed  project 
activities  is  equal  to  the  total  budget.  However,  for  activities  not  started,  the  earned  value 
is  equal  to  zero.  Objective  judgments  or  Performance  Measurement  Techniques  (PMT) 
refers  to  multiplying  the  budget  by  the  percentage  complete  to  get  the  earned  value.  The 
author  notes  that  work  perfonned  by  a  project  manager  and  quality  control  inspector  is 
referred  to  as  “level  of  effort”  and  value  is  as  budgeted.  Plus,  as  long  as  the  task  is 
completed  the  value  is  earned.  Figure  2  gives  the  Schedule  Variance  (SV)  minus  the 
difference  between  the  earned  value  and  the  budget  minus  the  Cost  Variance  (CV)  minus 
the  difference  between  the  earned  value  and  the  actual  costs. 

D.  TPM/EVM  DATA  DID  NOT  TRACK  WITH  QMM  DATA 

The  author  finds  an  inconclusive  correlation  between  QMM  and  TPM/EVM.  The 
data  given  in  Table  3  for  TPM/EVM  did  not  track  with  the  data  in  Tables  1  &2  for  QMM 
survey  questionnaires,  even  though  software  programs  A  &  B  provided  data  that  might 
lead  one  to  believe  that  there  is  a  correlation  between  QMM  and  TPM/EVM.  However, 
when  it  came  to  software  program  C  the  data  proved  to  be  inconclusive  for  a  correlation 
to  be  present.  The  QMM  survey  questionnaires  in  program  C  reflected  a  very  poor  score 
of  less  than  70%,  and  their  EVM/TPM  score  presented  an  acceptable  score  of  100%  or 
one  according  to  the  requirements  of  large  software  programs. 

The  author  noted  that  the  possible  causes  could  be  in  the  way  the  data  is  gathered, 
calculated  and  presented  for  TPM/EVM.  This  is  based  on  the  fact  that  TPM/EVM  data  is 

processed,  calculated  and  presented  during  meetings  that  are  held  weekly  by  Project 
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Managers  for  project  status.  Then  the  weekly  TPM/EVN  data  is  summarized  and 
presented  for  the  monthly  general  staff  presentation.  This  gives  the  Project  Managers 
time  so  they  can  make  adjustments  each  week  reflecting  an  acceptable  status  for  the 
monthly  general  staff  presentation.  Also,  new  requirements  or  changing  requirements 
constantly  being  placed  on  a  project  make  TPM/EVM  goals  difficult  at  best. 

The  author  noted  that  the  QMM  questionnaire  surveys  gives  better  details  of 
where  in  the  program  the  Program  Managers  are  possibly  having  difficulties.  QMM 
concentrates  on  four  basic  areas  in  the  questionnaire  surveys  such  as:  requirements 
management,  estimation/planning  management,  people  management  and  risk 
management.  The  TPM/EVM  data  gives  a  yes  or  no  answer  to  Program  Managers  on 
weekly  and  monthly  status  in  meeting  the  projects  technical  goals. 

E.  SUMMARY 

In  this  Section  III  the  author  presented  the  data  from  QMM  questionnaire  surveys 
to  measure  the  perception  of  program  managers  from  programs  A,  B  and  C  that  have  the 
responsibility  for  the  software  development  within  each  of  the  said  programs  for  the  U.S. 
Army.  The  author  then  gathered  TPM/EVM  data  using  a  metric  methodology  from  the 
same  programs  given  the  questionnaire  surveys,  and  developed  both  sets  of  data  tables 
for  review  of  possible  correlation  between  them.  The  author  noted  during  his  review  of 
these  data  tables  that  the  TPM/EVM  data  did  not  track  with  the  QMM  data  presented. 
Therefore  an  inclusive  correlation  between  QMM  and  TPM/EVM  was  presented. 

In  Section  IV  the  author  presents  Conclusions,  Recommendations,  and 
suggestions  for  Future  Work  based  on  his  findings. 
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IV.  CONCLUSIONS,  RECOMMENDATIONS  AND  FUTURE 

WORK 


A.  CONCLUSIONS 

This  thesis  provided  an  initial  evaluation  of  QMM  and  TPM  for  a  possible 
correlation  between  the  two  when  evaluating  software  management  and  technical 
performance  on  specific  software  programs.  The  software  programs  evaluated  varied 
considerably  and  played  a  significant  part  in  the  overall  success  of  a  larger  software 
program.  The  decisions  and  policies  that  program  managers  make  using  QMM  and 
EVM/TPM  could  provide  the  advantage  given  if  there  were  a  correlation  between  them. 
However,  Earned  Value  Management  (EVM)  did  not  track  with  QMM  as  test  data 
reflected.  Also,  the  author  notes  that  EVM/TPM  did  not  indicate  the  program  as 
successful  or  non-successful  as  QMM  provided  in  test  data  reflected  in  section  III 
showed  an  inconclusive  correlation  between  QMM  and  TPM/EVM. 

1.  QMM  Survey 

The  author  used  the  survey  format  provided  from  Martin  J,  Machniak  thesis, 
Development  of  a  Quality  Management  Metric  (QMM)  Measuring  Software  Program 
Management  Quality  December  1999.  The  format  of  the  QMM  survey,  and  the  individual 
questions  and  the  TPM  data  was  unchanged.  The  intent  of  this  thesis  was  to  find  out  if 
there  is  a  correlation  between  QMM  and  TPM.  This  thesis  achieved  the  goal  by  surveys 
and  EVM  data  taken  from  three  software  programs  found  on  a  major  Army  software 
program.  The  surveys  were  done  in  an  acceptable  amount  of  time  by  the  dedicated 
participants  in  programs  A,  B  and  C.  The  survey  completion  time  was  on  average 
approximately  90  minutes.  The  time  needed  to  take  the  survey  ranged  from  60  to  121 
minutes  approximately. 

2.  QMM  Scores 

All  three  programs  A,  B,  and  C,  having  QMM  percentage  scores  in  comparison  to 
the  individual  overall  program  success  scores  reflected  a  strong  correlation  between 
them.  However,  the  author  felt  that  the  program  managers  in  answering  the  survey 
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questions  needed  to  be  told  not  to  answer  the  survey  on  how  they  think  a  program  should 
be  managed  but  how  their  program  is  actually  run.  All  but  one  of  the  three  programs  had 
an  overall  success  score  of  seven,  which  corresponded  to  the  QMM  percentage  score  of 
70  percent  plus  or  minus  5  percent  points,  and  indicated  a  strong  correlation  according  to 
Martin  J.  Mackniak  QMM  standards  defined  in  his  thesis.  QMM  scores  compared  to  the 
QMM  Standard  scores  show  a  strong  correlation.  Two  out  of  three  survey  participants 
recorded  QMM  percentage  scores  within  5  points  of  the  mean  success  score  for  their 
respective  programs.  The  only  exception  was  program  C  where  it  fell  below  the  QMM 
standard  of  70  percent.  However,  this  program  was  dealing  with  very  immature  critical 
technologies  and  constant  change  in  requirements  of  which  the  program  manager  did  not 
have  control  over  such  changes. 

3.  TPM  DATA 

The  TPM  data  was  taken  for  a  ten  month  period  using  EVM  which  was  reported 
monthly  using  the  SPI  and  CPI  data.  The  performance  processes  of  the  TPM/EVM 
should  be  measured,  recorded,  and  scheduled  on  a  regular  basis  for  full  effectiveness  of 
the  process.  The  TPM  reports  that  are  not  reported  or  have  been  ignored  can  be 
considered  proof  that  the  TPM  process  is  not  being  used  and  is  a  possible  example  of  a 
non-valued  added  activity.  However,  if  indicators  for  ignoring  one  particular  TPM  are 
justified  then  it  should  be  closed  out  and  no  longer  reported.  The  reportable  SPI  and  the 
CPI  of  the  TPM  should  go  to  the  program  manager  and  IPT  keeping  everyone  fully  up¬ 
dated.  Three  out  of  the  three  programs  that  participated  recorded  EVM  percentage  scores 
within  the  set  standard.  In  the  EVM  the  standard  is  set  at  a  ratio  of  less  than  one  which 
implies  that  work  is  underachieving  against  the  plan,  and  above  one  implies  better  than 
planned,  where  100  percent  is  equal  to  one  This  is  the  acceptable  standard  set  for  large 
software  programs  within  the  U.S.  Anny. 

4.  TPM  DATA  DID  NOT  TRACK  QMM  DATA 

The  author  notes  that  in  the  case  of  program  C  which  had  constant  changes  in 
their  TPM/EVM  technical  requirements  gave  an  inconclusive  correlation  between  the 
QMM  data  and  TPM/EVM  data  in  section  III. 
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B.  RECOMMENDATIONS 

The  author  feels  that  by  using  both  the  TPM  (in  the  EVM  format),  and  the  QMM 
survey  questionnaires  as  possible  tools,  software  program  management  performance  can 
be  improved  through  complete  evaluation  of  EVM  data,  and  QMM  survey  questions  data. 
The  dichotomies  found  in  the  QMM  questionnaire  survey  by  participants  in  the  same 
software  program  during  the  same  time  period  need  to  be  discussed.  When  the  program 
manager  notes  a  change  in  the  EVM  data  where  the  TPM  does  not  meet  the  goals  set-up 
in  the  program,  the  QMM  survey  should  be  given  and  meetings  will  need  to  be  called  to 
discuss  differences  in  survey  questions,  and  change  in  TPM  status. 

1.  Evaluation  of  the  Survey  Sections 

As  new  changes  present  themselves  to  the  software  engineering  field,  new 
techniques,  followed  by  different  strategies  become  the  norm.  The  need  to  re-evaluate  the 
survey  sections  to  reflect  and  refine  the  need  for  better  software  quality  is  a  must.  The 
Program  Manager  can  read  the  survey  questions  having  a  view  of  the  past  and  present 
performance  of  their  software  program,  and  then  look  for  sections  that  score  the  lowest  as 
possible  areas  for  improvement. 

There  is  a  need  to  have  an  administrator  for  the  survey  to  help  with  the 
explanation  to  various  levels  of  program  management,  plus  to  help  uncover  any 
misperceptions  and  possible  pre-bias  that  can  exist  in  giving  survey  results. 

Also,  there  is  a  need  to  focus  on  survey  questionnaire  development  to  reflect  the 
continuous  changes  in  software  management  and  philosophy  as  an  on-going  process:  a) 
Concept  clarification  with  keeping  current  program  condition  as  the  object  of  each  survey 
question;  b)  Survey  question  replacement  to  reflect  new  trends  in  quality  software 
management;  c)  Giving  upper  program  management  the  option  to  change  the  sectional 
point  value  of  the  questions  in  order  to  help  determine  software  management  quality;  and 
d)  Keeping  the  survey  length  and  time  to  complete  to  a  minimum. 
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2. 


Evaluation  of  TPM/EVM  Sections 


Using  EVM/TPM  as  a  perfonnance  management  tool  can  ensure  a  project  is 
provided  the  best  possible  cost  and  schedule  impact  with  the  potential  to  offer 
organizations  significant  benefits  around  monitoring  and  managing  software  programs. 
However,  the  difficult  part  is  to  ensure  that  the  EVM  approach  focuses  squarely  on  the 
right  software  projects  and  monitors  them  at  the  right  level,  because  EVM  solutions  have 
failed  in  the  past  by  becoming  too  complicated,  and  therefore  cannot  be  maintained  by 
the  organization.  Having  experienced  EVM  and  TPM  personnel  on  staff  would  help  to 
establish  an  understanding  of  what  a  successful  EVM  solution  would  look  like  in  their 
dynamic  environment.  Also,  by  reviewing  the  core  topics  such  as:  a)  determining  the 
strategic  priorities  of  the  software  project;  b)  ensuring  that  the  EVM  analysis  focuses  on 
the  right  aspects  of  the  projects;  c)  designing  and  building  the  EVM  solution  that  suits 
your  project  needs  and  provides  the  appropriate  level  of  detail;  d)  reporting  EVM  finding 
in  a  way  that  everyone  can  understand  them;  e)  building  EVM  right  into  the  software 
program  budgeting  process. 

3.  TPM  and  QMM  Metrics 

In  this  thesis  the  author  provided  an  infonnal  verification,  validation,  and 
evaluation  of  only  three  software  programs  for  the  QMM  and  the  TPM/EVM  scores.  All 
three  of  these  programs  fell  under  the  Department  of  the  Army.  The  author  felt  that  due 
to  the  nature  in  which  software  programs  are  managed  in  this  environment  and  not  in  the 
civilian  work  place  many  more  software  programs  of  various  sizes  and  variety  need  to  be 
considered  before  establishing  a  statistically  sound  correlation  between  QMM  score  to 
overall  software  program  success  and  TPM  using  EVM  data.  The  metric  formulation  in 
scoring  will  require  possibly  different  coefficients,  and  should  be  based  on  the  software 
program  size,  complexity  and  environment,  whether  commercial  or  government. 

As  tools  for  measurement  improve  and  are  developed  and  what  is  considered  a 
quality  software  program  is  defined,  improvements  will  continue  to  come  forth  whether 
as  QMM  or  TPM.  The  author  noted  that  the  data  between  QMM  and  TPM/EVM,  even 
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though  it  gave  an  inconclusive  correlation  in  this  thesis,  may  be  developed  further  in  the 
future  as  we  search  how  to  make  a  successful  software  program. 

4.  Metric  Scoring  of  QMM  and  TPM/EVM 

In  this  thesis  informal  verification  and  validation,  provided  evaluation  for  three 
software  programs  for  a  correlation  between  QMM  and  TPM/EVM  of  which  all  of  the 
programs  were  Department  of  Defense  systems.  The  author  suggested  a  larger  sample 
should  be  taken  of  software  program  managers  and  key  team  members,  plus  a  greater 
variation  of  software  programs  need  to  be  evaluated  using  QMM  surveys  and  TPM/EVM 
data  before  a  well  defined  correlation  can  be  established  between  the  QMM  score  and  the 
TPM/EVM  data  with  respect  to  overall  success  of  a  software  program.  The  author  noted 
that  to  establish  a  template  to  evaluate  improvement  in  software  management 
performance  may  involve  repetition  of  a  computational  procedure.  This  replication  of  a 
cycle  of  operational  procedure  may  result  in  an  approximate  desired  outcome  that  is 
closer  to  the  set  goals.  The  formulation  of  coefficients  for  the  QMM  surveys  and 
EVM/TPM  data  may  need  to  vary  according  to  top  management’s  overall  program  goals, 
for  example,  taking  size,  use  and  complexity  of  the  software  being  developed  into 
consideration.  Software  Program  Managers  must  customize  their  approach  with  respect 
to  any  use  of  available  measurement  software  tools  such  as  QMM  surveys  and 
TMP/EVM  data,  keeping  project  goals  as  flexible  as  possible  for  directional  changes  by 
top  management. 

C.  FUTURE  WORK 

In  this  section  the  author  would  like  to  make  it  clear  to  the  reader  that  he  did  not 
address  the  relationship  between  Earned  Value  (EV)  and  QMM.  This,  the  author  has  left 
as  possible  future  work  as  stated  within  the  sections  of  this  thesis. 
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APPENDIX  A 


A.  PROGRAM  A  -  PROGRAM  MANAGER 
1.  QMM  Summary  Score  Sheet 


QMM  Scoresheet 

Part  One 

Part  Two 

Total 

Importance 

Weighted 

Category 

Score 

Score 

Score 

Coefficient 

Score 

Requirements 

Management 

a 

47 

e 

49 

96 

0.92 

= 

88.32 

Est./Planning 

Management 

D 

39 

D 

59 

98 

0.67 

= 

65.66 

People 

Management 

c 

45 

g 

46 

91 

1.86 

= 

169.26 

Risk  Management 

D 

55 

D 

46 

101 

0.55 

= 

55.55 

QMM  SCORE  378.79 


Max.  QMM  score  possible  528.00 

Min.  QMM  score  possible  -130.86 


QMM  percentage  score: 


77.35% 


Objective/Subjective  view  of  the  overall  success  of  program  A  on  a  scale  of  0  to  10 
(0  being  total  failure,  10  being  perfect  program  total  success) 

Survey  Participant:  Program  Manager 
Success  Score:  8 
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2.  Requirements  Management  Questionnaire  Responses 


No.  Requirements  Management  Questionnaire  -  Total:  Block  e _ Yes  No  N/A 


1 

PM  chose  to  have  a  formal  requirements  list 

X 

2 

Requirements  recorded  in  some  way 

X 

3 

Written  requirements  were  part  of  some  formal  document 

X 

4 

Written  requirements  were  informal 

X 

5 

At  least  some  requirements  were  oral  only 

X 

6 

All  stakeholders  were  identified 

X 

7 

All  stakeholders  participated  in  the  requirements  extraction 

X 

8 

Some  stakeholders  participated  in  the  requirements  extraction 

X 

9 

Management  extracted  requirements,  no  stakeholder  involvement 

X 

10 

Management  passed  requirements  to  development  team 

X 

11 

Stakeholders  not  involvved  in  Management  extraction,  but  approved 

X 

12 

Management  gets  inputs  from  stakeholders,  then  develops  requirements 

X 

13 

Developers  work  informally  with  users  to  arrive  at  requirements 

X 

14 

Same  as  13,  but  management  oversees  and  formalizes 

X 

|  If  a  waterfall  or  sequential  development  strategy:  1 

15 

All  requirements  complete  before  design 

X 

16 

Some  requirements  left  incomplete  prior  to  design 

X 

17 

Requirements  informal  prior  to  design  effort 

X 

18 

Requirements  serve  as  input 

X 

19 

Length  of  time  for  requirements  work  greater  than  development  work 

X 

20 

Requirements  developed  in  parallel  to  design 

X 

|  OR  If  a  prototype,  throwaway,  or  other  development  strategy:  j 

15 

Learn  about  requirements  through  development  efforts 

16 

No  coding  until  all  requirements  are  defined 

17 

Requirements  formal  prior  to  design  effort 

18 

Requirements  serve  as  output 

19 

Requirements  definition  work  in  parallel  to  development  efforts 

20 

Requirements  developed  in  parallel  to  design 

21 

Are  requirements  frozen  at  some  phase 

X 

22 

Change  management  exists 

X 

23 

Change  management  is  formal 

X 

24 

Project  strategy  is  consistent  throughout  development 

X 

25 

Requirements  are  updated 

X 

26 

Configuration  Management  (CM)  exists 

X 

27 

CM  is  formal 

X 

28 

Requirements  are  testable 

X 

29 

Requirements  testing  considered/implemented  during  extraction 

X 

30 

Requirements  testing  plan  exists 

X 

31 

Requirements  testing  is  formal 

X 

32 

All  requirements  have  priorities 

X 

33 

All  requirements  must  be  implemented 

X 

34 

Requirements  are  tested 

X 

35 

All  requirements  are  equally  important 

X 

36 

At  least  some  requirements  have  priorities 

X 

37 

All  requirements  are  traceable 

X 

38 

Traceability  not  important 

X 

39 

Each  requirement  has  an  author 

X 

40 

Who  authored  requirement  is  not  important 

X 

41 

Initial  set  of  requirements  to  be  implemented,  no  requirements  creep 

X 

42 

Structured  and  tracked  changes  to  requirements  only 

X 

43 

Change  is  inevitable,  changes  allowed  at  all  times 

X 

44 

Change  is  inevitable,  but  changes  limited 

X 

45 

Requirements  control  funding 

X 

46 

Requirements  history  kept 

X 

47 

Baseline  established  for  requirements  at  some  point  prior  to  develop 

X 

TOTAL  SCORING 

EM 

■za 

36 


3.  Estimation/Planning  Questionnaire  Responses 


No.  Estimation/Planning  Questionnaire  -  Total:  Block  f _ Yes  No  N/A 


1 

A  volume  product  metric  used  (LOC,  #  of  files,  #  of  screens,  pages  of  doc) 

X 

2 

Measure  used  for  various  product  elements  (modules,  components,  CSCI) 

X 

3 

Product  measures  made  by  phase  (amt  at  implementation,  LOC  changed  at  unit  test) 

X 

4 

Other  product  attributes  measured  (FP,  throughput,  mem  cap,  cyclomatic  complexity) 

X 

5 

Product  matrics  tracked  and  updated  hroughout  program  execution 

X 

6 

Event  count  process  metric  used  (#  defects  in  test,  reqmt  changes,  milestones  met) 

X 

7 

Time  measure  process  metric  used  (cycle  time) 

X 

8 

Process  metrics  tracked  and  updated  throughout  program  execution 

X 

9 

Program  cost  estimations  made  from  product  or  process  metrics 

X 

10 

Program  cost  extimations  tracked  and  updated  to  reflect  progress/changes 

X 

11 

Factor  analysis  performed  on  program 

X 

12 

Program's  primary  purpose,  including  major  functions  and  deliverables  known 

X 

13 

Work  breakdown  structure  developed 

X 

14 

Task  estimated  with  realistic  expectations  of  productivity  probabilities 

X 

15 

Schedules  developed  based  on  realistic  expectations 

X 

16 

Schedules  tracked  and  updated  based  on  new  information 

X 

17 

Detailed  activity  lists  used  for  clearly  defined  completed/not  completed  tasks 

X 

18 

Quality  assurance  plan  or  similar  to  aid  in  detecting  defects  early  in  program 

X 

19 

COCOMO  estimates  performed 

X 

20 

CSCI  dearly  defined  and  tasked 

X 

21 

Estimates  completed  ad  hoc 

X 

22 

Gantt  charts  used  and  updated 

X 

23 

Resource  estimations  (working  hrs,  job  categories,  task  activities)  done 

X 

24 

Earned  value  established 

X 

25 

Earned  value  tracked  throughout  program 

X 

26 

Quality  expectations  established  for  product  with  users  and  stakeholders 

X 

27 

Critical  path  for  program  tasks  developed  and  tracked 

X 

28 

Measure  of  effectiveness  (MOE)  or  Figure  of  merit  established  and  tracked 

X 

29 

Estimates  are  updated  routinely 

X 

30 

Schedules  are  updated  routinely 

X 

31 

Estimations  are  made  by  program  management  (top-down) 

X 

32 

Estimateions  are  made  by  program  team  members  (bottom-up) 

X 

33 

Automated  program  tracking  used 

X 

34 

PM  usually  thorough  in  tracking  and  reporting  schedules  and  financials 

X 

35 

WBS  developed  only  as  data  call 

X 

36 

Earned  value  used  to  track  program  progress 

X 

37 

PM  insists  on  prioritizing  work  reduction  as  schedule/funding  compromised  by  stakeholders 

X 

38 

Estimations  are  done  using  both  top  down  and  bottoms  up  approaches 

X 

39 

All  program  team  members  involved  in  planning  process 

X 

40 

Flardware  also  considered  in  estimaation  process 

X 

41 

Program  history  compiled 

X 

42 

System  upgrades  (SCR)  software  changes  requests  estimated  individually 

X 

43 

Management  duties  apart  of  each  team  member's  responsibilities 

X 

44 

PM  dictates  schedules  to  program  team 

X 

45 

Code  reviews  planned  in  schedule 

X 

46 

Defined  tangible  milestones  established  for  program  tasks 

X 

47 

Test  planning  done  at  the  start  of  the  program 

X 

48 

Estimations  are  completed  by  those  performing  the  tasks 

X 

49 

Sensitivity  analysis  performed  for  program  choices 

X 

50 

Software  deployment  planning  completed 

X 

TOTAL  SCORING 
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4.  People  Management  Questionnaire  Responses 


No.  People  Management  Questionnaire  -  Total  Block  g _ Yes  No  N/A 


1 

PM  is  accessible  in  person  by  each  team  member 

X 

2 

PM  is  accessible  via  email  (memo,  letter)  by  each  team  member 

X 

3 

PM  is  accessible  via  phone  by  each  team  member 

X 

4 

PM  not  only  considers  a  person's  suitability,  not  also  desire  to  be  on  a  team 

X 

5 

PM  consults  with  each  team  member  regarding  their  career  goals 

X 

6 

PM  regularly  holds  meetings  to  inform  team  of  program  progress 

X 

7 

PM  solicits  opinions  from  team  members  before  making  decisions 

X 

8 

PM  lets  teams  make  decisions  affecting  their  work 

X 

9 

PM  freuently  makes  decisions  without  any  consultation  with  members 

X 

10 

PM  understands  the  technology/language  of  the  program 

X 

11 

PM  is  able  to  communicate  with  other  the  technical  issues  in  the  program 

X 

12 

PM  prioritized  problems  or  conflicts  within  the  program 

X 

13 

PM  assists  team  members  in  developing/advising  of  career  path 

14 

PM  empowers  program  members  to  recommend  hiring  new  team  members 

X 

15 

PM  empowers  program  members  to  recommend  firings  of  other  members 

X 

16 

PM  specifically  assigns  work  to  each  program  member 

X 

17 

PM  sets  communication  protocol 

X 

18 

PM  allows  unrestricted  communications 

X 

19 

PM  encourages  or  requires  training  for  each  individual 

X 

20 

PM  takes  control  in  difficult/roblem  areas 

X 

21 

PM  looks  ahead  to  new  programs,  new  upgrades  of  existing  program 

X 

22 

PM  maintains  regular  communications  with  all  stakeholders 

X 

23 

PM  maintains  regular  communications  with  users 

X 

24 

PM  encourages  program  team  communication  with  users 

X 

25 

PM  encourages  program  team  communication  with  stakeholders 

X 

26 

PM  facilitates  horizontal  communication  within  program 

X 

27 

PM  facilitates  communication  during  integration 

X 

28 

PM  holds  meetings  without  clear  objectives 

X 

29 

PM  must  approve  all  decisions  within  the  program 

X 

30 

PM  must  approve  all  interactions  with  stakeholders 

X 

31 

PM  must  approve  all  interactions  with  users 

X 

32 

PM  makes  all  presentations  to  stakeholders/users 

X 

33 

PM  is  considered  "flexible"  in  terms  of  program  members  personal  issues 

X 

34 

PM,  at  least  occasionally,  schedules/promotes  outside  work  team  activities 

X 

35 

PM  is  readily  willing  to  listen  to  program  prblems  and  complaints 

X 

36 

PM  takes  action  to  resolve  program  problems  and  complaints 

X 

37 

PM  is  generally  respected  by  stakeholders,  users,  and  organization 

X 

38 

PM  sometimes  fails  to  grasp  important  technical  issues  in  program 

X 

39 

PM  recruits  program  team  members  from  outside  organization 

X 

40 

PM  participates  in  technical  reviews 

X 

41 

Program  personnel  have  clearly  defined  specific  tasks 

X 

42 

Although  individual's  tasks  are  specific,  each  exposed  to  the  "bigger  picture" 

X 

43 

PM  has  clearly  defined  his/her  expectations  for  each  individual 

X 

44 

PM  delegation  of  duties  is  usually  seemless  in  execution 

X 

45 

PM  acts  as  facilitator  to  solving  personnel  conflicts 

X 

46 

PM  attempts  to  motivate  individuals  on  the  program  team 

X 

47 

PM  clearly  spearates  technical  from  managerial  roles  for  individuals 

X 

48 

PM  directs  how  he/she  expects  the  task  to  be  accomplished 

X 

49 

PM  directs  what  needs  to  be  done,  but  does  not  direct  how 

X 

50 

PM  attempts  to  spotlight  individuals  in  the  program  for  positive  exposure 

X 

TOTAL  SCORING 
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5. 


Risk  Management  Questionnaire  Responses 


No.  Risk  Management  Questionnaire  -  Block  h _ Yes  No  N/A 


1 

Risk  Management  (RM)  is  specifically  an  activity  in  the  program 

X 

2 

RM  is  formal  and  documented 

X 

3 

A  specific  RM  Ian  exists 

X 

4 

RM  is  required  in  the  program,  but  not  used  during  the  program 

X 

5 

RM  is  done  prior  to  the  program  execution 

X 

6 

RM  is  done  by  an  outside  entity  to  the  development 

X 

7 

RM  is  done  internally  only 

X 

8 

RM  is  both  internally  performed  and  externally  assessed 

X 

9 

RM  planning  occurs  during  or  after  major  milestones  in  the  program 

X 

10 

Risk  Assessment  is  only  a  management  function 

X 

11 

RM  is  informal  or  non  existent 

X 

12 

There  is  a  RM  plan,  but  it  is  not  updated  or  tracked 

X 

13 

Risks  are  only  generalized 

X 

14 

Each  risk  is  delineated 

X 

15 

Each  risk  has  a  consequence 

X 

16 

Each  risk  has  a  likelihood  rating  of  some  sort 

X 

17 

Each  risk  has  a  mitigation  strategy 

X 

18 

Risk  Management  is  automated 

X 

19 

Risks  are  tracked 

X 

20 

X 

21 

Regret  analysis  performed 

X 

22 

RM  drives  decisions  in  the  program 

X 

23 

Risks  have  probabilities 

X 

24 

Risk  Management  is  ad  hoc 

X 

25 

RM  information  is  shared  with  all  stakeholders  (as  appropriate) 

X 

26 

Risks  are  weighed  relative  to  other  program  risks 

X 

27 

Risk  Assessment  is  a  program  team  activity 

X 

28 

Risk  Assessment  done  prior  to  program  start 

X 

29 

Risk  Assessment  includes  personnal  risk 

X 

30 

RM  uses  tools,  but  depends  on  human  decisions 

X 

31 

Risk  assessment  includes  cost  risks 

X 

32 

Risk  Assessment  includes  schedule  risks 

X 

33 

Risk  Assessment  includes  technology  risks 

X 

34 

Risk  Assessment  is  briefed  organization  structure  above  program  manager 

X 

35 

Risk  Assessment  includes  requirements  risks 

X 

36 

Risk  Assessment  includes  user  risks  (too  little  involvement  of  user) 

X 

37 

Risk  Assessment  includes  documentation  risks 

X 

38 

Risk  Assessment  includes  integration  risks 

X 

39 

Risk  Assessment  includes  interface  risks  (non-standard) 

X 

40 

Risk  Assessment  includes  continuing  requirements  change  (feature  creep) 

X 

41 

Risk  Assessment  includes  dependent  projects/programs  risks 

X 

42 

Documentation  proof  exists  to  demonstrate  following  risk  management  plan 

X 

43 

High  rish  have  measured  tracking  (high  profile  status) 

X 

44 

Organizational  history  used  to  search  for  risks 

X 

45 

Other  organizational  checklists  used  for  risk  assessment 

X 

46 

Internal  organizational  checklists  used  for  risk  assessment 

X 

47 

Risk  Assessment  information  contributed  to  internal  or  other  database 

X 

48 

Risk  Assessment  includes  internal  organization  risks 

X 

49 

Risk  Assessment  includes  stakeholder  risks 

X 

50 

No  risk  management  needed;  program  is  straightforward  &  understood 

X 

TOTAL  SCORING 
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6.  Pair  Choices  Responses 

Pair  choice  section  ONE:  (Requirements  Management)  choose  most  applicable  term  of  the  two  for  each  row  (page  1  of  2): 


formal  requirement  list 

X 

informal  requirement  list 

written  requirements 

X 

oral  requirements 

requirements  informal,  but  recorded 

requirements  not  recorded 

X 

requirements  as  part  of  an  SRS  (or  other  formal  repository) 

requirements  informally  recorded 

X 

requirements  taken  as  is  from  customer 

look  to  reformulate,  interview  in-depth,  or  otherwise  re-validate 

X 

only  one  development  strategy  used 

strategies  not  consistent,  used  at  different  times 

X 

stakeholders  as  part  of  requirements  development 

stakeholders  approving  requirements  after  formulated  by  development  team 

X 

requirements  are  testable 

requirements  have  no  test  plans 

X 

informal  test  plan  or  no  test  plan 

formal  test  plan 

X 

test  team  involved  with  requirements 

X 

no  test  team  input  or  plans  during  requirements  development 

only  a  percentage  of  requirements  present  in  baseline 

baseline  must  contain  all  requirements 

X 

requirements  documentation  has  hierarchical  structure 

all  requirements  must  be  implemented 

X 

requirements  have  listed  responsible  party 

X 

requirements  origin  not  important 

requirements  documentation  have  versions 

X 

no  requirements  history 

requirements  have  specific  attribute  values 

X 

requirements  all  rank  evenly 

funding  controls  requirements  definition 

requirements  definition  controls  funding 

X 

reqquirements  are  top  down 

X 

requirements  are  bottom  up 

users/stakeholders  are  identified  and  interviewed  (market  survey) 

no  special  consideration  to  identify  users/stakeholders 

X 

each  requirement  has  a  singular  concept 

some  requirements  are  compound  statements 

X 

requirements  definition  minimized  when  funding  short 

program  scope  may  reduce,  but  requirements  definition  completed 

X 

requirements  extraction  has  formal  process 

X 

requirements  extraction  ad  hoc 

change  procedures  formal 

X 

change  procedures  ad  hoc 

users/stakeholders  somehow  involved  in  requirements  definition 

X 

program  team  only  involved  in  requirement  definition 

management  sets  requirements  for  developers 

developers  at  least  partially  involved  in  setting  requirements 

X 

requirements  changed  at  least  once  since  baseline  established  prior  to  new  version 

X 

requirements  in  baseline  has  not  changed  prior  to  new  version  or  upgrade 

no  ranking  of  requirements 

requirements  have  priorities  assigned 

X 

use-case  diagrams  (or  other  models  or  scenario  developments) 

X 

no  models  used  for  requirements  extraction 

requirements  changes  informal 

requirements  changes  formal 

X 

plan  to  "freeze"  requirements  at  some  designated  milestone 

X 

no  provision  for  "freezing"  requirements 

requirements  must  be  traceable 

X 

origin  of  requirements  not  important 

requirements  must  be  testable 

X 

system  developed  must  be  testable 

test  plans  to  determine  requirements  implemented 

X 

no  test  plans  needed  for  requirements  verification 

requirements  have  priorities  in  implementation 

all  requirements  must  be  implemented 

X 

some  requirements  have  multiple  statements  or  ideas 

X 

one  idea,  one  statement  per  requirement 

Requirements  Management  (page  1  of  2)  score 
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Pair  choice  section  ONE:  (Requirements  Management)  choose  most  applicable  term  of  the  two  for  each  row  (page  2  of  2): 


ANSWER  THIS  BLOCK  OF  QUESTIONS  ONLY  IF  A  SEQUENTIAL  OR  WATERFALL  APPROACH  IS  USED  FOR  DEVELOPMENT  (Requirements  page  2  of  2) 

requirements  first,  then  initial  development  work 

initial  development  work  then  requirements 

X 

requirements  documentation  driving  development 

requirements  documentation  developed  in  parallel/after  development 

X 

user  feedback  considered  during  development 

after  development  starts,  user  feedback  serves  as  input  to  new  work 

X 

change  management  procedures  used  strictly 

X 

change  management  procedures  as  guidance  only 

design  decisions  prior  to  or  in  parallel  to  requirrements  development 

X 

design  decisions  only  after  approved  requirements  stabilized 

requirements  summarized  wht  we  have  developed 

requirements  are  the  blueprint  for  development 

X 

length  of  time  for  requirements  work  greater  than  development  work 

X 

length  of  time  for  requirements  work  less  than  development  work 

X 

requirements  have  design  detail 

no  design  detail  in  requirements 

X 

requirements  creep  to  be  avoided 

requirements  creep  o.k.,  but  need  to  be  controlled 

X 

freeze  requirements  at  some  point 

requirements  are  fluid  throughout  development 

formal  change  procedure 

X 

informal  change  procedure 

change  management  plan 

X 

no  change  management  plan 

requirements  ambiguity  always  present  to  some  extent 

X 

requirements  ambuiguity  unacceptable  at  any  level 

testing  considered  up  frornt  during  requirements  determination 

testing  considered  down  the  line  during  development 

X 

requirements  development  team  members  different  from  implementation 

those  working  on  requirements,  work  on  implementation 

X 

start  implementation  as  early  as  possible  to  help  define  requirements 

requirements  must  be  defined  prior  to  any  implementation  work 

X 

ANSWER  THIS  BLOCK  OF  QUESTIONS  ONLY  IF  A  PROTOTYPING,  THROWAWAY,  SYNCHRONIZE  &  STABILIZE,  OR  OTHER  STRATEGY  USED 

develop  prototype,  then  determine  requirements 

determine  requirements  prior  to  any  development  work 

requirements  testing  done  after  each  iteration 

no  testing 

individual  changes  as  necessary 

only  block  changes  made 

development  team  decides  on  changes  after  iteration 

users  involved  with  changes 

changes  based  on  feedback  only  from  user  for  correction  of  problems 

changes  to  upgrade  system  and  correct  problems 

funding  controls  changes  and  change  procedures 

changes  control  funding 

requirements  documentation  finalized  prior  to  development 

requirements  fluid  throughout  development  (only  freeze  at  end) 

requirements  test  plans  completed  prior  to  development 

requirements  test  plans  completed  after  development 

requirements  first,  then  initial  development  work 

initial  development  work  then  requirements 

use  development  effort  to  learn  more  about  requirements 

define  all  requirements  prior  to  coding  anything 

requirements  ambiguity  always  present  to  some  extent 

requirements  ambiguity  unacceptable  at  any  level 

requirements  have  design  detail 

no  design  detail  in  requirements 

user  feedback  considered  during  development 

after  development  starts,  user  feedback  serves  as  input  to  new  work 

get  something  to  users  as  soon  as  possible  for  evaluation 

make  sure  it  is  complete  before  releasing 

management  dictates  requirements 

development  team  visually  represent  requirements  through  rapid  prototyping 

new  requirements  allowed  after  initial  requirements  defined 

new  requirements  not  allowed 

Requirements  Management  (pg  2  of  2)  score  [12]  +pg  1  score  [35]  =  TOTAL  SCORE  [47]  Enter  on  QMM  scoresheet  blk  a. 
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Pair  choice  section  TWO:  (Estimation/Planning  Management)  choose  most  applicable  term  of  the  two  for  each  row  (page  1  of  2): 


at  least  one  estimation  method  used  in  program 

X 

no  estimates 

formal  derivation  of  product  metric  for  estimation  of  size 

X 

ad  hoc  size  estimation 

ad  hoc  process  evaluation 

formal  derivation  of  at  lest  one  process  metric 

X 

develop  work  breakdown  structure  (WBS) 

X 

assign  work  as  needs  arise 

estimates  are  developed  to  fulfill  a  data  call  only 

use  estimates  to  plan  program 

X 

use  estimates  to  sell  program  only 

estimates  are  useful  to  the  project  tema  for  planning  purposes 

X 

resource  evaluations  made  for  program 

no  resource  evaluation  for  planning 

use  both  bottom  up  &  top  down  for  estimate,  use  one  stakeholders  like 

X 

use  both  bottom  up  &  top  down  and  evaluate  significant  differences 

estimates  made  and  not  updated 

X 

estimates  updated  throughout  program 

resources  estimations  used  to  adjust  product  size  estimate 

X 

estimations  made  irregardless  of  resources  available 

estimations  made  to  fit  budget 

budget  made  from  estimations 

X 

estimations  compromised  to  get  program 

rather  risk  loss  of  program  than  compromise  confident  estimations 

X 

cycle  time  estimations 

no  cycle  time  estimations 

X 

event  count  estimations 

no  event  count  estimations 

X 

lines  of  code  (LOC)  estimation 

X 

no  LOC  estimation 

function  pont  (FP)  estimation 

X 

no  FP  estimation 

estimates  by  algorithmic  methods 

estimates  by  analogy 

X 

expert  judgement  for  estimates 

X 

ad  hoc  estimates 

estimates  by  algorithmic  methods 

X 

ad  hoc  estimates 

expert  judgement  for  estimates 

X 

estimates  by  analogy 

ad  hoc  estimates 

estimates  by  analogy 

X 

bottom  up  estimates 

X 

expert  judgement 

top  down  estimates 

X 

expert  judgement 

ad  hoc  estimates 

any  other  estimate  process 

X 

fuzzy  logic  estimating  method 

X 

no  formal  estimation  methodology 

WBS  development  from  estimates 

X 

WBS  development  in  parallel  or  prior  to  estimation  completion 

critical  path  of  program  determined 

tasks  developed  but  no  path  is  identified 

X 

estimators  are  program  team  members 

X 

estimators  are  outside  program  team 

management  only  on  estimations 

all  team  members  involved  in  estimation  process 

X 

estimates  updated  at  reviews 

no  updates  of  estimates 

X 

estimates  updated  at  reviews 

X 

estimates  constantly  updates  (in  between  reviews,  to) 

estimate  procedures  stay  the  same 

X 

estimate  procedures  change 

stakeholders  are  part  of  estimation  process 

stakeholders  brief  estimations  after  completion 

X 

estimates  are  used  beyond  initial  selling  of  program 

X 

estimates  are  one  time  events,  used  for  a  specific  purpose  once 

WBS  has  objective  measure  of  completeness 

important  to  have  WBS  as  guide,  not  rigid  implementation 

X 

Estimation/Planning  Management  (page  1  of  2)  score 
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Pair  choice  section  TWO:  (Estimation/Planning  Management)  choose  most  applicable  term  of  the  two  for  each  row  (page  2  of  2): 


life  cycle  estimates 

estimates  for  program  initiation  only 

X 

system  upgrades  (SCR)  software  change  requests  estimated  individually 

X 

systems  upgrades  estimated  as  whole 

estimates  for  on-gong  resources  needed  to  maintain  s/w 

estimates  for  maintenance  not  done 

X 

informal  re-estimates  during  development 

X 

formal  re-estimates  at  pre-defined  milestones 

formal  re-estimates  when  amendment  changing  the  system  is  introduced 

X 

informal  re-estimates  when  amendment  changing  the  system 

person  in-charge  of  estimation  walks  in  a  managers  office  to  get  an  opinion 

X 

meeting(s)  organized  for  purpose  of  performing  cost  estimations 

factor  analysis  prior  to  commencement  of  program 

none  done 

X 

change  control  procedures  set  in  place 

X 

no  set  procedures 

elapsed  time  and  actual  work  time  estimates 

one  or  the  other  or  neither 

X 

no  schedule  created 

X 

scheudle  created 

schedule  not  updated 

X 

schedule  updated 

schedule  followed 

X 

schedule  not  followed 

tasks  identification  arises  as  program  progresses 

X 

detailed  level  tasks  identified  prior  to  program  initiation 

scope  of  program  understood  by  all 

scope  not  explicitly  defined 

X 

quality  factors  and  criteria  identified 

X 

no  explicit  quality  factors  defined 

no  project  tracking  tools  used 

project  tracking  tools  used 

X 

CSCIs  identified  and  tasked 

X 

CSCIs  not  explicitly  identified 

expectations  are  managed  via  estimations 

X 

estimations  are  made  to  fit  preconceived  expectations 

no  cost  schedule  developed 

cost  schedule  developed 

no  resource  schedule  developed 

X 

resource  schedule  developed 

team  members,  management  know  at  any  time  if  in  budget  &  schedule 

exact  budget  &  schedule  status  somewhat  unclear  to  at  least  some 

X 

individual  program  phases  are  estimated 

X 

only  top  level  program  estimated 

stakeholders/users  emphasis  understood-quick  to  field  or  all  complete 

program  management  sets  delivery  tradeoffs  without  outside  input 

testing  planned  with  initial  program  planning 

X 

testing  not  in  initial  planning 

documentation  not  considered  ininitial  planning 

X 

documentation  part  of  initial  planning 

hardware  considered  in  estimations 

X 

software  only  considered 

no  formal  schedule/cost  tracking 

X 

formal  procedures  established  for  tracking  cost  and  schedule 

earned  value  set  up 

X 

earned  value  not  used 

estimations  omit  documentation  planning 

X 

documentation  in  estimates 

training  omitted  in  estimates 

X 

training  part  of  estimates 

earned  value  set  up,  but  not  tracked 

earned  value  tracked 

X 

detailed  planning  done  with  incomplete  set  of  requirements 

X 

detailed  planning  done  with  detailed  set  of  requirements 

complete  infrastructure  support  mechanism  understood  for  estimations 

no  consideration  of  infrastructure  done  for  estimations 

X 

team  possibilities  considered  for  planning  of  program 

X 

no  consideration  for  outside  teaming  possibilities 

work  breakdown  structure  (WBS)  set  up 

no  WBS  completed 

X 

Estimation/Planning  Management  (pg  2  of  2)  score  [14]  +pg  1  score 

[25]  = 

TOTAL  SCORE  [39]  Enter  on  QMM  scoresheet  blk  b. 

43 


Pair  choice  section  THREE:  (People  Management)  choose  most  applicable  term  of  the  two  for  each  row  (page  1  of  2): 
Human  Resources 


program  team  members  have  clearly  deined,  segmented  roles 

X 

work  responsibilities  are  shared 

formal  team  building  procedures  are  used 

no  formal  team  building  emphasized 

X 

program  manager  flexible  regarding  work  hours 

X 

program  manager  maintains  strict  standards  for  work  hours 

big  picture  conveyed  to  all  team  members  by  program  management 

X 

program  management  focuses  on  the  partitioned  tasks  with  team 

people  issues  dealt  with  primarily  through  indirect  methods  (email,  memo,  etc) 

people  issues  dealt  with  primarily  through  direct  methods  (face-to-face) 

X 

training  is  required  and  planned  on  a  regular  basis 

training  is  ad  hoc 

X 

each  team  member  is  educated  on  and  understands  overall  program  and  their  roles 

team  members  only  know  their  respective  areas 

X 

consideration  for  team  members'  career  goals  are  reflected  in  assignments 

X 

team  members  must  adapt  to  tasks  that  are  assigned 

team  members  assignments  and  responsibilities  are  mostly  dictated  by  PM 

assignments  and  responsibilities  are  discussed  and  agreed  upon  with  PM 

X 

management  leads  in  problem  solving 

management  facilitates  and  lets  team  lead  in  problem  solving 

X 

management  welcomes  problems  as  challenges  and  opportunities 

X 

management  views  problems  as  obstacles  and  grounds  for  punishment 

team  members  participate  in  performance  evaluations  of  peers 

Personnel  evaluations  are  strictly  PM  responsibility 

X 

management  reinforcement  feedback  sparse  and  inconsistent,  if  any 

X 

management  provides  timely  reinforcement  feedback  for  positive  behaviors 

management  provides  basic  needs  of  office  facilities  fairly  well 

X 

office  facilities  are  a  drawback  to  working  in  the  program 

working  conditions  are  fairly  comfortable,  time  off  policy  fairly  good 

X 

working  conditions  and  time  off  policy  is  inconsistent  and  difficult  at  times 

Communication: 


communications  primarily  written  (email) 

communications  primarily  verbal  (face-to-face) 

X 

detailed  instructions:  oral  presentation,  follow-up  email 

X 

email  only 

formal  communication  protocol 

informal  communications 

X 

external  vertical  communications  restricted 

X 

external  vertical  communication  allowed 

coders  notebook  weekly  accomplishment  reports  required 

X 

not  required 

user-coder  relationship  established,  encouraged,  and  mediated 

user-coder  interaction  minimized 

X 

meetings  structured  to  minimize  waster  time 

X 

meetings  unstructured  and  open  ended 

meetings  have  agenda,  objectives,  and  conclude  with  action  items 

X 

meeting  agenda  fluid  and  open  ended 

program  management  and  coder  communication  face  to  face 

X 

program  management  and  coder  communication  primarily  email 

program  team  updated  regularly  regarding  organizational  &  program  status 

meetings  infrequently  scheduled 

X 

open  communications  is  encouraged 

X 

communication  hrough  chain  of  command  only  is  encouraged 

program  manager  accessible  for  discussions 

X 

program  manager  difficult  to  get  an  appointment  to  see 

program  management  (PM)  is  viewed  as  separate  from  team 

PM  mixes  with  team  frequently 

X 

management  regularly  holds  team  meetings 

X 

meetings  are  sporadic 

meetings  are  structured  with  definite  goals  and  objectives 

X 

meetings  are  informal 

program  management  is  generally  easy  to  reach  and  talk  to 

X 

PM  is  usually  hard  to  get  a  hold  of  and  difficult  to  talk  to 

team-program  manager  relationship  adult-adult 

X 

team-program  management  relationship  parent-child 

schedules  are  spontaneous  and  poorly  communicated 

schedules  must  be  fixed  and  rigidly  followed  and  formally  reported 

X 

work  is  seen  as  complex  processes  involving  team  working  together 

X 

work  broken  into  pieces  with  minimal  team  member  interaction 

action  items  often  is  poorly  disseminated  and  usually  not  followed  through 

action  items  communicated  and  followed  through  thoroughly 

X 

team  members  require  frequent  clarifications  by  PM  for  assigned  tasks 

team  members  rarly  require  clarifications  by  PM  for  assigned  tasks 

X 
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Pair  choice  section  THREE:  (People  Management)  choose  most  applicable  term  of  the  two  for  each  row  (page  2  of  2): 
Leadership: 


long  range  organizational  vision 

short  tern  program  and  immediate  w  ork  focus 

X 

lead  through  personal  attention  to  others 

action-oriented  leadership  approach 

X 

run  as  much  of  the  organization  as  possible 

let  team  make  decisions  as  much  as  possible 

X 

direct  and  domineering  style 

X 

encourage  independence  in  others 

traditional  leaders  respect  hierarchy 

do  w  hat  needs  to  be  done 

X 

w  in  cooperation  rather  than  demand  it 

X 

tough-minded  with  others 

act  strongly  and  forcefully  in  the  field  of  ideas 

X 

prefer  to  lead  other  independent  types  while  seeking  auto  no  my  for  self 

consults  w  ith  team  members  to  find  solutions  to  problems 

X 

consults  team  members  to  get  validation  of  FMs  predetermined  solutions 

keep  people  w  ell  informed 

X 

only  as  much  know  ledge  as  necessary  for  their  w  ork 

make  things  happen  by  focusing  on  the  immediate  problems 

X 

long  range  focus  and  de-emphasize  current  problems 

manage  others  loosely  and  prefer  minimal  supervision 

follow  traditional  procedures  and  rules  conscientiously 

X 

leadership,  management  decisions  exclusively  by  program  management 

program  management  makes  decisions  but  gets  inputs  from  team 

X 

team-program  manager  relationship  adult-adult 

X 

team-program  management  relationship  parent-child 

program  management  makes  decisions  but  gets  inputs  from  team 

X 

all  program  team  members  responsible  for  program  decisions 

w  hen  a  problem  arises:  management  takes  over  to  solve  it 

X 

management  lets  the  team  solve  the  problems 

leadership  is  do  as  1  say,  not  do  as  1  do 

X 

leadership  by  example 

program  expectation  not  influenced  by  PM 

program  expectation  managed  by  PM 

X 

PM  gives  freedom  to  team,  but  has  no  mentoring  for  members  (abdication) 

X 

PM  empow  ers  teams  by  mentoring  members  to  be  leaders 

promgram  management  w  aits  and  sees  w  hat  happens  then  plans 

management  plans  far  in  advance 

X 

program  management  is  constantly  reacting  to  emergencies 

X 

management  is  one  step  ahead  of  problems 

facilitative  approach  to  solving  problems 

take  charge  readily  and  often 

X 

program  management  is  complex,  takes  much  time  to  understand 

X 

management  is  simple,  easy  to  figure  out 

program  management  prefers  to  plunge  right  in 

X 

takes  time  to  separate  things  to  be  done  and  order  of  doing  them 

program  management  reacts  spur  of  the  moment 

X 

methodically  follows  plans 

Technical  Competency  of  the  Program  Manager: 


PM  has  technical  experience  particular  to  the  particular  s/w  program 

X 

PM  relies  on  team  members  solely 

PM  participates  in  technical  review  s 

X 

PM  only  in  non-technical  reviews 

PM  participates  in  making  technical  decisions  when  problems  arise 

X 

PM  delegates  technical  questions 

PM  does  not  get  involved  discussing  technical  options 

PM  contributes  to  technical  options  being  discussed 

X 

PM  does  not  review  technical  options  and  decisions 

PM  reviews  technical  options  and  decisions 

X 

PM  actively  attempts  to  keep  up-to-date  with  current  techno  logy  and  standards 

PM  is  removed  from  cutting  edge  technology  issues 

X 

PM  receives  technical  periodicals  and  occasionally  references  applicable  articles 

X 

PM  doesn't  read  periodicals  nor  reference  current  articles  to  team 

PM  doesn't  have  technical  background  (or  education) 

PM  has  technical  background  (or  education) 

X 

team  members  avoid  PM  w  hen  they  need  technical  advice 

team  members  generally  consider  talking  to  PM  regarding  technical  issues 

X 

HR[9]  +  Comm.  [17]  +  Leadership  [11]  +  Tech.  Competency  [8]  =  People  Mgmt.  score  [45]  Enter  on  QMM  scoresheet  blk  c. 
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Pair  choice  section  FOUR:  (Risk  Management(RM))  choose  most  applicable  term  of  the  two  for  each  row  (page  1  of  2): 


RM  is  formal  and  documented 

X 

RM  is  informal,  if  at  all 

a  risk  management  plan  exists 

X 

no  risk  management  plan  is  developed 

RM  is  more  of  a  data  call  than  a  useful  document 

RM  drives  decisions  on  the  program 

X 

RM  is  done  prior  to  the  program  beginning 

RM  is  done  prior  and  during  program  execution 

X 

RM  is  only  done  during  the  program  execution 

RM  is  done  prior  and  during  program  execution 

X 

risks  are  generalized  through  the  whole  program 

risks  are  categorized 

X 

risk  management  is  done  internally,  only 

an  outside  organization  also  contributes  to  the  RM  process 

X 

risk  is  a  management  function 

risk  is  a  program  team  function 

X 

risks  are  precisely  articulated 

risks  are  generalized,  if  at  ail 

each  risk  has  a  consequence 

X 

consequences  are  generalized,  if  at  all 

a  mitigation  strategy  is  completed  for  each  risk 

X 

mitigation  strategy  is  generalized,  if  at  all 

contingency  plans  are  developed  for  a  RM  plan 

X 

contingency  plans  are  ad  hoc  as  problems  arise  in  the  program 

risks  are  anticipated 

if  problems  arise,  management  will  deal  with  it 

X 

the  program  doesn't  have  any  risk 

programs  that  do  not  have  risk,  have  problems 

X 

risk  management  is  automated 

risk  management  may  use  tools,  but  depend  on  human  input 

X 

risks  are  assigned  probabilities 

probabilities  are  not  relevant  for  RM 

X 

all  risks  are  potential  problems,  relative  priorities  for  risks  are  not  useful 

risks  are  weighed  relative  to  other  program  risks  and  thus  prioritized 

X 

risk  management  information  is  only  shared  internally 

risk  management  information  is  shared  with  all  stakeholders 

X 

risk  analysis  uses  ordinal  rankings 

risk  analysis  uses  actual  measurements  with  a  mathematical  model 

X 

regret  analysis  used 

X 

no  regret  analysis  done 

attach  probabilities  to  future  events 

X 

no  probabilities  associated  with  future  events 

assessing  risks  with  mechanical  meethods 

risks  should  be  compared  to  other  risks  and  sorted 

X 

risk  status  tracked 

X 

not  tracked 

technical  risks  examined 

X 

no  technical  risks  examined 

process  risks  examined 

X 

no  process  risks  examined 

product  risks  examined 

X 

no  product  risks  examined 

stakeholder/user  risks  examined 

X 

no  examination  of  stakeholder/user  risks 

checklists  used  to  identify  risks 

no  checklists  used 

X 

risks  are  tracked 

no  tracking  or  monitoring  of  risks 

X 

each  risk  has  an  impact 

X 

no  impact  analysis  of  risk 

each  risk  has  a  mitigation  plan 

X 

no  individual  risk  mitigation 

risks  monitored  by  priority 

X 

no  special  attention  to  track  higher  priority  risks 

X 

risk  assessment  is  formalized 

X 

no  formal  risk  assessment 

risk  control  is  formalized 

X 

no  formal  risk  control 

integration  risks  not  considered 

integration  risks  examined 

X 

Risk  Management  (page  1  of  2)  score 


I  30  I 
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Pair  choice  section  FOUR:  (Risk  Management(RM))  choose  most  applicable  term  of  the  two  for  each  row  (page  2  of  2): 


risks  to  cost 

X 

no  cost  risks  examined 

unforeseen  risks  have  occurred  in  program 

X 

any  risk  that  came  up  had  been  identified  previously 

personnel  risks  examined 

X 

no  personnel  risks  examined 

estimation  risks  examined 

X 

no  estimation  risks  examined 

planning  risks  examined 

X 

no  planning  risks  examined 

requirements  risks  examined 

X 

no  requirements  risks  examined 

resource  risks  examined 

X 

no  resource  risks  examined 

risk  management  plan  updated  regularly 

X 

no  regular  risk  management  plan  updates 

risks  charted 

X 

risks  not  charted 

performance  risks  examined 

X 

performance  risks  not  examined 

program  management  self  risks  examined 

X 

no  program  management  risks  examined 

risk  from  program  constraints  examined 

X 

no  program  constraint  risks  examined 

each  category  of  risks  are  prioritized 

X 

no  prioritization 

each  category  of  risks  are  evaluated  for  impact 

X 

no  impact  analysis  performed 

each  category  of  risks  have  control  strategy 

X 

no  control  strategy 

documentation  risks  examined 

X 

no  documentation  risks  examined 

regret  matrix  tracked 

no  regret  matrix  or  not  tracked 

X 

communication  of  risk  activities  are  facilitated 

X 

no  facilitation  or  promotion  of  communication  of  risk  activities 

X 

taxonomy-based  questionnaire  used  to  identify  risks 

taxonomy-based  questionnaire  not  used 

X 

associated  hardware  risks  examined 

X 

no  consideration  for  hardware  risks 

integration  risks  examined 

X 

integration  risks  not  examined 

communication  risks  examined 

X 

communication  risks  not  examined 

leadership  risks  examined 

X 

leadership  risks  not  considered 

risk  avoidance  considered  for  certain  risks 

X 

risk  avoidance  not  considered  for  risks 

risk  documentation  forms  used 

no  risk  documentation  forms  used 

X 

dependency  risks  examined 

X 

no  dependency  risks  examined 

alternatives  like  risk  avoidance  considered  for  high  risk  items 

X 

no  consideration  of  risk  avoidance 

documented  risk  statements  use  a  condition-consequence  type  format 

condition-consequence  of  risk  statements  not  clearly  defined 

X 

no  assignment  of  ownership  of  risk  mitigation  action 

X 

each  risk  mitigation  action  is  assigned  to  an  individual  for  resolution 

calculation  of  risk  exposure  made  (probability  X  loss,  for  each  risk) 

no  risk  exposure  calculations 

X 

oral  communication  of  risks  only 

X 

risks  written  in  a  way  that  communicates  nature  and  status  of  factors 

triggers  used  to  quantify  risk  conditions  present 

X 

risk  conditions  present  are  all  subjective 

X 

risk  "czar"  in  program  for  monitoring  risks 

no  special  positions/responsibilities  for  risk  monitoring 

X 

post-program  review  completed  (scheduled)  for  unanticipated  problems  ID 

no  post-program  reviews  completed  or  scheduled 

X 

no  schedule  risks  examined 

risks  to  schedule  investigated 

X 

Risk  Management  (pg  2  of  2)  score  [25]  +pg  1  score  [30]  =  TOTAL  SCORE  [55]  Enter  on  QMM  scoresheet  blk  d. 
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B.  PROGRAM  A  -  ASSOCIATE 

1.  QMM  Summary  Score  Sheet 


QMM  Scoresheet 

Part  One 

Part  Two 

Total 

Importance 

Weighted 

Category 

Score 

Score 

Score 

Coefficient 

Score 

Requirements 

Management 

a 

44 

e 

51 

95 

0.92 

= 

87.4 

Est./Planning 

Management 

D 

52 

D 

47 

99 

0.67 

= 

66.33 

People 

Management 

c 

54 

g 

45 

99 

1.86 

= 

184.14 

Risk  Management 

D 

55 

D 

43 

98 

0.55 

= 

53.9 

QMM  SCORE  391.77 


Max.  QMM  score  possible  528.00 

Min.  QMM  score  possible  -130.86 

QMM  percentage  score:  79.32% 


Objective/Subjective  view  of  the  overall  success  of  program  A  on  a  scale  of  0  to  10 
(0  being  total  failure,  10  being  perfect  program  total  success) 

Survey  Participant:  Associate 

Success  Score:  8 
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2.  Requirements  Management  Questionnaire  Responses 


No.  Requirements  Management  Questionnaire  -  Total:  Block  e  Yes  No  N/A 


1 

PM  chose  to  have  a  formal  requirements  list 

X 

2 

Requirements  recorded  in  some  w  ay 

X 

3 

Written  requirements  were  part  of  some  formal  document 

X 

4 

Written  requirements  were  informal 

X 

5 

At  least  some  requirements  were  oral  only 

X 

6 

All  stakeholders  were  identified 

X 

7 

All  stakeholders  participated  in  the  requirements  extraction 

X 

8 

Some  stakeholders  participated  in  the  requirements  extraction 

X 

9 

Management  extracted  requirements,  no  stakeholder  involvement 

X 

10 

Management  passed  requirements  to  development  team 

X 

11 

Stakeholders  not  involvved  in  Management  extraction,  but  approved 

12 

Management  gets  inputs  from  stakeholders,  then  develops  requirements 

X 

13 

Developers  w  ork  informally  w  ith  users  to  arrive  at  requirements 

X 

14 

Same  as  13,  but  management  oversees  and  formalizes 

X 

If  a  waterfall  or  sequential  development  strategy:  | 

15 

All  requirements  complete  before  design 

X 

16 

Some  requirements  left  incomplete  prior  to  design 

X 

17 

Requirements  informal  prior  to  design  effort 

X 

18 

Requirements  serve  as  input 

X 

19 

Length  of  time  for  requirements  work  greater  than  development  w ork 

X 

20 

Requirements  developed  in  parallel  to  design 

X 

OR  If  a  prototype ,  throwaway,  or  other  development  strategy:  j 

15 

Learn  about  requirements  through  development  efforts 

X 

16 

No  coding  until  all  requirements  are  defined 

17 

Requirements  formal  prior  to  design  effort 

18 

Requirements  serve  as  output 

19 

Requirements  definition  work  in  parallel  to  development  efforts 

X 

20 

Requirements  developed  in  parallel  to  design 

X 

21 

Are  requirements  frozen  at  some  phase 

X 

22 

Change  management  exists 

X 

23 

Change  management  is  formal 

X 

24 

Ftoject  strategy  is  consistent  throughout  development 

X 

25 

Requirements  are  updated 

X 

26 

Configuration  Management  (CM)  exists 

X 

27 

CM  is  formal 

X 

28 

Requirements  are  testable 

X 

29 

Requirements  testing  considered/implemented  during  extraction 

X 

30 

Requirements  testing  plan  exists 

X 

31 

Requirements  testing  is  formal 

X 

32 

All  requirements  have  priorities 

X 

33 

All  requirements  must  be  implemented 

X 

34 

Requirements  are  tested 

X 

35 

All  requirements  are  equally  important 

X 

36 

At  least  some  requirements  have  priorities 

X 

37 

All  requirements  are  traceable 

X 

38 

Traceability  not  important 

X 

39 

Each  requirement  has  an  author 

X 

40 

Who  authored  requirement  is  not  important 

X 

41 

Initial  set  of  requirements  to  be  implemented,  no  requirements  creep 

X 

42 

Structured  and  tracked  changes  to  requirements  only 

X 

43 

Change  is  inevitable,  changes  allowed  at  all  times 

X 

44 

Change  is  inevitable,  but  changes  limited 

X 

45 

Requirements  control  funding 

X 

46 

Requirements  history  kept 

X 

47 

Baseline  established  for  requirements  at  some  point  prior  to  develop 

X 

TOTAL  SCORING 

m 

~2~ 
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3.  Estimation/Planning  Questionnaire  Responses 


No.  Estimation/Planning  Questionnaire  -  Total:  Block  f  Yes  No  N/A 


1 

A  volume  product  metric  used  (LOC,  #  of  files,  #  of  screens,  pages  of  doc) 

X 

2 

Measure  used  for  various  product  elements  (modules,  components,  CSCI) 

X 

3 

Product  measures  made  by  phase  (amt  at  implementation,  LOC  changed  at  unit  test) 

X 

4 

Other  product  attributes  measured  (FP,  throughput,  mem  cap,  cyclomatic  complexity) 

X 

5 

Product  matrics  tracked  and  updated  hroughout  program  execution 

X 

6 

Event  count  process  metric  used  (#  defects  in  test,  reqmt  changes,  milestones  met) 

X 

7 

Time  measure  process  metric  used  (cycle  time) 

X 

8 

Process  metrics  tracked  and  updated  throughout  program  execution 

X 

9 

Program  cost  estimations  made  from  product  or  process  metrics 

X 

10 

Program  cost  extimations  tracked  and  updated  to  reflect  progress/changes 

X 

11 

Factor  analysis  performed  on  program 

X 

12 

Program's  primary  purpose,  including  major  functions  and  deliverables  known 

X 

13 

Work  breakdown  structure  developed 

X 

14 

Task  estimated  with  realistic  expectations  of  productivity  probabilities 

X 

15 

Schedules  developed  based  on  realistic  expectations 

X 

16 

Schedules  tracked  and  updated  based  on  new  information 

X 

17 

Detailed  activity  lists  used  for  clearly  defined  completed/not  completed  tasks 

X 

18 

Quality  assurance  plan  or  similar  to  aid  in  detecting  defects  early  in  program 

X 

19 

COCOMO  estimates  performed 

X 

20 

CSCI  clearly  defined  and  tasked 

X 

21 

Estimates  completed  ad  hoc 

X 

22 

Gantt  charts  used  and  updated 

X 

23 

Resource  estimations  (working  hrs,  job  categories,  task  activities)  done 

X 

24 

Earned  value  established 

X 

25 

Earned  value  tracked  throughout  program 

X 

26 

Quality  expectations  established  for  product  with  users  and  stakeholders 

X 

27 

Critical  path  for  program  tasks  developed  and  tracked 

X 

28 

Measure  of  effectiveness  (MOE)  or  Figure  of  merit  established  and  tracked 

X 

29 

Estimates  are  updated  routinely 

X 

30 

Schedules  are  updated  routinely 

X 

31 

Estimations  are  made  by  program  management  (top-down) 

X 

32 

Estimateions  are  made  by  program  team  members  (bottom-up) 

X 

33 

Automated  program  tracking  used 

X 

34 

PM  usually  thorough  in  tracking  and  reporting  schedules  and  financials 

X 

35 

WBS  developed  only  as  data  call 

X 

36 

Earned  value  used  to  track  program  progress 

X 

37 

PM  insists  on  prioritizing  work  reduction  as  schedule/funding  compromised  by  stakeholders 

X 

38 

Estimations  are  done  using  both  top  down  and  bottoms  up  approaches 

X 

39 

All  program  team  members  involved  in  planning  process 

X 

40 

Hardware  also  considered  in  estimaation  process 

X 

41 

Program  history  compiled 

X 

42 

System  upgrades  (SCR)  software  changes  requests  estimated  individually 

X 

43 

Management  duties  apart  of  each  team  member's  responsibilities 

X 

44 

PM  dictates  schedules  to  program  team 

X 

45 

Code  reviews  planned  in  schedule 

X 

46 

Defined  tangible  milestones  established  for  program  tasks 

X 

47 

Test  planning  done  at  the  start  of  the  program 

X 

48 

Estimations  are  completed  by  those  performing  the  tasks 

X 

49 

Sensitivity  analysis  performed  for  program  choices 

X 

50 

Software  deployment  planning  completed 

X 

TOTAL  SCORING 
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4.  People  Management  Questionnaire  Responses 


No.  People  Management  Questionnaire  -  Total:  Block  g  Yes  No  N/A 


1 

PM  is  accessible  in  person  by  each  team  member 

X 

2 

PM  is  accessible  via  email  (memo,  letter)  by  each  team  member 

X 

3 

PM  is  accessible  via  phone  by  each  team  member 

X 

4 

PM  not  only  considers  a  person's  suitability,  not  also  desire  to  be  on  a  team 

X 

5 

PM  consults  with  each  team  member  regarding  their  career  goals 

X 

6 

PM  regularly  holds  meetings  to  inform  team  of  program  progress 

X 

7 

PM  solicits  opinions  from  team  members  before  making  decisions 

X 

8 

PM  lets  teams  make  decisions  affecting  their  work 

X 

9 

PM  freuently  makes  decisions  without  any  consultation  with  members 

X 

10 

PM  understands  the  technology/language  of  the  program 

X 

11 

PM  is  able  to  communicate  with  other  the  technical  issues  in  the  program 

X 

12 

PM  prioritized  problems  or  conflicts  within  the  program 

X 

13 

PM  assists  team  members  in  developing/advising  of  career  path 

X 

14 

PM  empowers  program  members  to  recommend  hiring  new  team  members 

X 

15 

PM  empowers  program  members  to  recommend  firings  of  other  members 

X 

16 

PM  specifically  assigns  work  to  each  program  member 

X 

17 

PM  sets  communication  protocol 

X 

18 

PM  allows  unrestricted  communications 

X 

19 

PM  encourages  or  requires  training  for  each  individual 

X 

20 

PM  takes  control  in  difficult/roblem  areas 

X 

21 

PM  looks  ahead  to  new  programs,  new  upgrades  of  existing  program 

X 

22 

PM  maintains  regular  communications  with  all  stakeholders 

X 

23 

PM  maintains  regular  communications  with  users 

X 

24 

PM  encourages  program  team  communication  with  users 

X 

25 

PM  encourages  program  team  communication  with  stakeholders 

X 

26 

PM  facilitates  horizontal  communication  within  program 

X 

27 

PM  facilitates  communication  during  integration 

X 

28 

PM  holds  meetings  without  clear  objectives 

X 

29 

PM  must  approve  all  decisions  within  the  program 

X 

30 

PM  must  approve  all  interactions  with  stakeholders 

X 

31 

PM  must  approve  all  interactions  with  users 

X 

32 

PM  makes  all  presentations  to  stakeholders/users 

X 

33 

PM  is  considered  "flexible"  in  terms  of  program  members  personal  issues 

X 

34 

PM,  at  least  occasionally,  schedules/promotes  outside  work  team  activities 

X 

35 

PM  is  readily  willing  to  listen  to  program  prblems  and  complaints 

X 

36 

PM  takes  action  to  resolve  program  problems  and  complaints 

X 

37 

PM  is  generally  respected  by  stakeholders,  users,  and  organization 

X 

38 

PM  sometimes  fails  to  grasp  important  technical  issues  in  program 

X 

39 

PM  recruits  program  team  members  from  outside  organization 

X 

40 

PM  participates  in  technical  reviews 

X 

41 

Program  personnel  have  clearly  defined  specific  tasks 

X 

42 

Although  individual's  tasks  are  specific,  each  exposed  to  the  "bigger  picture" 

X 

43 

PM  has  clearly  defined  his/her  expectations  for  each  individual 

X 

44 

PM  delegation  of  duties  is  usually  seemless  in  execution 

X 

45 

PM  acts  as  facilitator  to  solving  personnel  conflicts 

X 

46 

PM  attempts  to  motivate  individuals  on  the  program  team 

X 

47 

PM  clearly  spearates  technical  from  managerial  roles  for  individuals 

X 

48 

PM  directs  how  he/she  expects  the  task  to  be  accomplished 

X 

49 

PM  directs  what  needs  to  be  done,  but  does  not  direct  how 

X 

50 

PM  attempts  to  spotlight  individuals  in  the  program  for  positive  exposure 

X 

TOTAL  SCORING 
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5. 


Risk  Management  Questionnaire  Responses 


No.  Risk  Management  Questionnaire  -  Total:  Block  h  Yes  No  N/A 


1 

Risk  Management  (RM)  is  specifically  an  activity  in  the  program 

X 

2 

RM  is  formal  and  documented 

X 

3 

A  specific  RM  Ian  exists 

X 

4 

RM  is  required  in  the  program,  but  not  used  during  the  program 

X 

5 

RM  is  done  prior  to  the  program  execution 

X 

6 

RM  is  done  by  an  outside  entity  to  the  development 

X 

7 

RM  is  done  internally  only 

X 

8 

RM  is  both  internally  performed  and  externally  assessed 

X 

9 

RM  planning  occurs  during  or  after  major  milestones  in  the  program 

X 

10 

Risk  Assessment  is  only  a  management  function 

X 

11 

RM  is  informal  or  non  existent 

X 

12 

There  is  a  RM  plan,  but  it  is  not  updated  or  tracked 

X 

13 

Risks  are  only  generalized 

X 

14 

Each  risk  is  delineated 

X 

15 

Each  risk  has  a  consequence 

X 

16 

Each  risk  has  a  likelihood  rating  of  some  sort 

X 

17 

Each  risk  has  a  mitigation  strategy 

X 

18 

Risk  Management  is  automated 

X 

19 

Risks  are  tracked 

X 

20 

21 

Regret  analysis  performed 

X 

22 

RM  drives  decisions  in  the  program 

X 

23 

Risks  have  probabilities 

X 

24 

Risk  Management  is  ad  hoc 

X 

25 

RM  information  is  shared  with  all  stakeholders  (as  appropriate) 

X 

26 

Risks  are  weighed  relative  to  other  program  risks 

X 

27 

Risk  Assessment  is  a  program  team  activity 

X 

28 

Risk  Assessment  done  prior  to  program  start 

X 

29 

Risk  Assessment  includes  personnal  risk 

X 

30 

RM  uses  tools,  but  depends  on  human  decisions 

X 

31 

Risk  assessment  includes  cost  risks 

X 

32 

Risk  Assessment  includes  schedule  risks 

X 

33 

Risk  Assessment  includes  technology  risks 

X 

34 

Risk  Assessment  is  briefed  organization  structure  above  program  manager 

X 

35 

Risk  Assessment  includes  requirements  risks 

X 

36 

Risk  Assessment  includes  user  risks  (too  little  involvement  of  user) 

X 

37 

Risk  Assessment  includes  documentation  risks 

X 

38 

Risk  Assessment  includes  integration  risks 

X 

39 

Risk  Assessment  includes  interface  risks  (non-standard) 

X 

40 

Risk  Assessment  includes  continuing  requirements  change  (feature  creep) 

X 

41 

Risk  Assessment  includes  dependent  projects/programs  risks 

X 

42 

Documentation  proof  exists  to  demonstrate  following  risk  management  plan 

X 

43 

High  rish  have  measured  tracking  (high  profile  status) 

X 

44 

Organizational  history  used  to  search  for  risks 

X 

45 

Other  organizational  checklists  used  for  risk  assessment 

X 

46 

Internal  organizational  checklists  used  for  risk  assessment 

X 

47 

Risk  Assessment  information  contributed  to  internal  or  other  database 

X 

48 

Risk  Assessment  includes  internal  organization  risks 

X 

49 

Risk  Assessment  includes  stakeholder  risks 

X 

50 

No  risk  management  needed;  program  is  straightforward  &  understood 

X 

TOTAL  SCORING 
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6.  Pair  Choices  Responses 

Pair  choice  section  ONE:  (Requirements  Management)  choose  most  applicable  term  of  the  two  for  each  row  (page  1  of  2): 


formal  requirement  list 

X 

informal  requirement  list 

w  ritten  requirements 

X 

oral  requirements 

requirements  informal,  but  recorded 

X 

requirements  not  recorded 

requirements  as  part  of  an  SRS  (or  other  formal  repository) 

X 

requirements  informally  recorded 

requirements  taken  as  is  from  customer 

X 

look  to  reformulate,  interview  in-depth,  or  otherwise  re-validate 

only  one  development  strategy  used 

strategies  not  consistent,  used  at  different  times 

stakeholders  as  part  of  requirements  development 

stakeholders  approving  requirements  after  formulated  by  development  team 

requirements  are  testable 

X 

requirements  have  no  test  plans 

informal  test  plan  or  no  test  plan 

formal  test  plan 

test  team  involved  w  ith  requirements 

X 

no  test  team  input  or  plans  during  requirements  development 

only  a  percentage  of  requirements  present  in  baseline 

baseline  must  contain  all  requirements 

requirements  documentation  has  hierarchical  structure 

all  requirements  must  be  implemented 

requirements  have  listed  responsible  party 

requirements  origin  not  important 

requirements  documentation  have  versions 

X 

no  requirements  history 

requirements  have  specific  attribute  values 

requirements  all  rank  evenly 

funding  controls  requirements  definition 

X 

requirements  definition  controls  funding 

reqquirements  are  top  dow  n 

X 

requirements  are  bottom  up 

users/stakeholders  are  identified  and  interviewed  (market  survey) 

X 

no  special  consideration  to  identify  users/stakeholders 

each  requirement  has  a  singular  concept 

some  requirements  are  compound  statements 

requirements  definition  minimized  when  funding  short 

program  scope  may  reduce,  but  requirements  definition  completed 

requirements  extraction  has  formal  process 

X 

requirements  extraction  ad  hoc 

change  procedures  formal 

X 

change  procedures  ad  hoc 

users/stakeholders  somehow  involved  in  requirements  definition 

X 

program  team  only  involved  in  requirement  definition 

management  sets  requirements  for  developers 

developers  at  least  partially  involved  in  setting  requirements 

requirements  changed  at  least  once  since  baseline  established  prior  to  newversion 

X 

requirements  in  baseline  has  not  changed  prior  to  newversion  or  upgrade 

no  ranking  of  requirements 

X 

requirements  have  priorities  assigned 

use-case  diagrams  (or  other  models  or  scenario  developments) 

no  models  used  for  requirements  extraction 

requirements  changes  informal 

requirements  changes  formal 

plan  to  "freeze"  requirements  at  some  designated  milestone 

X 

no  provision  for  "freezing"  requirements 

requirements  must  be  traceable 

X 

origin  of  requirements  not  important 

requirements  must  be  testable 

X 

system  developed  must  be  testable 

test  plans  to  determine  requirements  implemented 

X 

no  test  plans  needed  for  requirements  verification 

requirements  have  priorities  in  implementation 

all  requirements  must  be  implemented 

some  requirements  have  multiple  statements  or  ideas 

X 

one  idea,  one  statement  per  requirement 

Requirements  Management  (page  1  of  2)  score  |  31 
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Pair  choice  section  ONE:  (Requirements  Management)  choose  most  applicable  term  of  the  two  for  each  row  (page  2  of  2): 


j  ANSWER  THIS  BLOCK  OF  QUESTIONS  ONLY  IF  A  SEQUENTIAL  OR  WA  TERFALL  APPROACH  IS  USED  FOR  DEVELOPMENT  (Requirements  page  2  of  2)  | 

requirements  first,  then  initial  development  work 

initial  development  work  then  requirements 

requirements  documentation  driving  development 

requirements  documentation  developed  in  parallel/after  development 

user  feedback  considered  during  development 

after  development  starts,  user  feedback  serves  as  input  to  new  work 

change  management  procedures  used  strictly 

change  management  procedures  as  guidance  only 

design  decisions  prior  to  or  in  parallel  to  requirrements  development 

design  decisions  only  after  approved  requirements  stabilized 

requirements  summarized  wht  we  have  developed 

requirements  are  the  blueprint  for  development 

length  of  time  for  requirements  work  greater  than  development  work 

length  of  time  for  requirements  work  less  than  development  work 

requirements  have  design  detail 

no  design  detail  in  requirements 

requirements  creep  to  be  avoided 

requirements  creep  o.k.,  but  need  to  be  controlled 

freeze  requirements  at  some  point 

requirements  are  fluid  throughout  development 

formal  change  procedure 

informal  change  procedure 

change  management  plan 

no  change  management  plan 

requirements  ambiguity  always  present  to  some  extent 

requirements  ambuiguity  unacceptable  at  any  level 

testing  considered  up  frornt  during  requirements  determination 

testing  considered  down  the  line  during  development 

requirements  development  team  members  different  from  implementation 

those  working  on  requirements,  work  on  implementation 

start  implementation  as  early  as  possible  to  help  define  requirements 

requirements  must  be  defined  prior  to  any  implementation  work 

|  ANSWER  THIS  BLOCK  OF  QUESTIONS  ONLY  IF  A  PROTOTYPING,  THROWAWAY,  SYNCHRONIZE  &  STABILIZE,  OR  OTHER  STRATEGY  USED  1 

develop  prototype,  then  determine  requirements 

determine  requirements  prior  to  any  development  work 

X 

requirements  testing  done  after  each  iteration 

no  testing 

X 

individual  changes  as  necessary 

X 

only  block  changes  made 

development  team  decides  on  changes  after  iteration 

X 

users  involved  with  changes 

changes  based  on  feedback  only  from  user  for  correction  of  problems 

changes  to  upgrade  system  and  correct  problems 

X 

funding  controls  changes  and  change  procedures 

X 

changes  control  funding 

requirements  documentation  finalized  prior  to  development 

X 

requirements  fluid  throughout  development  (only  freeze  at  end) 

requirements  test  plans  completed  prior  to  development 

X 

requirements  test  plans  completed  after  development 

requirements  first,  then  initial  development  work 

X 

initial  development  work  then  requirements 

X 

use  development  effort  to  learn  more  about  requirements 

X 

define  all  requirements  prior  to  coding  anything 

requirements  ambiguity  always  present  to  some  extent 

X 

requirements  ambiguity  unacceptable  at  any  level 

requirements  have  design  detail 

X 

no  design  detail  in  requirements 

user  feedback  considered  during  development 

X 

after  development  starts,  user  feedback  serves  as  input  to  new  work 

get  something  to  users  as  soon  as  possible  for  evaluation 

X 

make  sure  it  is  complete  before  releasing 

management  dictates  requirements 

X 

development  team  visually  represent  requirements  through  rapid  prototyping 

new  requirements  allowed  after  initial  requirements  defined 

X 

new  requirements  not  allowed 

Requirements  Management  (pg  2  of  2)  score  [13]  +pg  1  score  [31]  =  TOTAL  SCORE  [44]  Enter  on  QMM  scoresheet  blk  a. 
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Pair  choice  section  TWO:  (Estimation/Planning  Management)  choose  most  applicable  term  of  the  two  for  each  row  (page  1  of  2): 


at  least  one  estimation  method  used  in  program 

X 

no  estimates 

formal  derivation  of  product  metric  for  estimation  of  size 

ad  hoc  size  estimation 

ad  hoc  process  evaluation 

formal  derivation  of  at  lest  one  process  metric 

develop  work  breakdown  structure  (WBS) 

X 

assign  work  as  needs  arise 

estimates  are  developed  to  fulfill  a  data  call  only 

use  estimates  to  plan  program 

use  estimates  to  sell  program  only 

estimates  are  useful  to  the  project  tema  for  planning  purposes 

resource  evaluations  made  for  program 

X 

no  resource  evaluation  for  planning 

use  both  bottom  up  &  top  down  for  estimate,  use  one  stakeholders  like 

use  both  bottom  up  &  top  down  and  evaluate  significant  differences 

estimates  made  and  not  updated 

estimates  updated  throughout  program 

resources  estimations  used  to  adjust  product  size  estimate 

X 

estimations  made  irregardless  of  resources  available 

estimations  made  to  fit  budget 

budget  made  from  estimations 

estimations  compromised  to  get  program 

rather  risk  loss  of  program  than  compromise  confident  estimations 

cycle  time  estimations 

X 

no  cycle  time  estimations 

event  count  estimations 

X 

no  event  count  estimations 

lines  of  code  (LOC)  estimation 

X 

no  LOC  estimation 

function  pont  (FP)  estimation 

no  FP  estimation 

estimates  by  algorithmic  methods 

estimates  by  analogy 

expert  judgement  for  estimates 

X 

ad  hoc  estimates 

estimates  by  algorithmic  methods 

X 

ad  hoc  estimates 

expert  judgement  for  estimates 

estimates  by  analogy 

ad  hoc  estimates 

estimates  by  analogy 

bottom  up  estimates 

X 

expert  judgement 

top  down  estimates 

X 

expert  judgement 

ad  hoc  estimates 

any  other  estimate  process 

fuzzy  logic  estimating  method 

no  formal  estimation  methodology 

WBS  development  from  estimates 

WBS  development  in  parallel  or  prior  to  estimation  completion 

critical  path  of  program  determined 

X 

tasks  developed  but  no  path  is  identified 

estimators  are  program  team  members 

X 

estimators  are  outside  program  team 

management  only  on  estimations 

all  team  members  involved  in  estimation  process 

estimates  updated  at  reviews 

X 

no  updates  of  estimates 

estimates  updated  at  reviews 

estimates  constantly  updates  (in  between  reviews,  to) 

estimate  procedures  stay  the  same 

estimate  procedures  change 

stakeholders  are  part  of  estimation  process 

stakeholders  brief  estimations  after  completion 

estimates  are  used  beyond  initial  selling  of  program 

X 

estimates  are  one  time  events,  used  for  a  specific  purpose  once 

WBS  has  objective  measure  of  completeness 

X 

important  to  have  WBS  as  guide,  not  rigid  implementation 

Estimation/Planning  Management  (page  1  of  2)  score 

29 
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Pair  choice  section  TWO:  (Estimation/Planning  Management)  choose  most  applicable  term  of  the  two  for  each  row  (page  2  of  2): 


life  cycle  estimates 

X 

estimates  for  program  initiation  only 

system  upgrades  (SCR)  software  change  requests  estimated  individually 

X 

systems  upgrades  estimated  as  w  hole 

estimates  for  on-gong  resources  needed  to  maintain  s/w 

X 

estimates  for  maintenance  not  done 

informal  re-estimates  during  development 

formal  re-estimates  at  pre-defined  milestones 

X 

formal  re-estimates  w  hen  amendment  changing  the  system  is  introduced 

X 

informal  re-estimates  w  hen  amendment  changing  the  system 

person  in-charge  of  estimation  w  alks  in  a  managers  office  to  get  an  opinion 

X 

meeting(s)  organized  for  purpose  of  performing  cost  estimations 

factor  analysis  prior  to  commencement  of  program 

none  done 

X 

change  control  procedures  set  in  place 

X 

no  set  procedures 

elapsed  time  and  actual  w  ork  time  estimates 

one  or  the  other  or  neither 

X 

no  schedule  created 

scheudle  created 

X 

schedule  not  updated 

schedule  updated 

X 

schedule  followed 

X 

schedule  not  followed 

tasks  identification  arises  as  program  progresses 

X 

detailed  level  tasks  identified  prior  to  program  initiation 

scope  of  program  understood  by  all 

X 

scope  not  explicitly  defined 

quality  factors  and  criteria  identified 

no  explicit  quality  factors  defined 

X 

no  project  tracking  tools  used 

project  tracking  tools  used 

X 

CSCIs  identified  and  tasked 

CSCIs  not  explicitly  identified 

X 

expectations  are  managed  via  estimations 

X 

estimations  are  made  to  fit  preconceived  expectations 

no  cost  schedule  developed 

cost  schedule  developed 

X 

no  resource  schedule  developed 

resource  schedule  developed 

X 

team  members,  management  know  at  any  time  if  in  budget  &  schedule 

exact  budget  &  schedule  status  somew  hat  unclear  to  at  least  some 

X 

individual  program  phases  are  estimated 

X 

only  top  level  program  estimated 

stakeholders/users  emphasis  understood-quick  to  field  or  all  complete 

X 

program  management  sets  delivery  tradeoffs  w  ithout  outside  input 

testing  planned  w  ith  initial  program  planning 

X 

testing  not  in  initial  planning 

documentation  not  considered  ininitial  planning 

X 

documentation  part  of  initial  planning 

hardware  considered  in  estimations 

X 

software  only  considered 

no  formal  schedule/cost  tracking 

formal  procedures  established  for  tracking  cost  and  schedule 

X 

earned  value  set  up 

X 

earned  value  not  used 

estimations  omit  documentation  planning 

X 

documentation  in  estimates 

training  omitted  in  estimates 

X 

training  part  of  estimates 

earned  value  set  up,  but  not  tracked 

X 

earned  value  tracked 

detailed  planning  done  w  ith  incomplete  set  of  requirements 

X 

detailed  planning  done  w  ith  detailed  set  of  requirements 

complete  infrastructure  support  mechanism  understood  for  estimations 

X 

no  consideration  of  infrastructure  done  for  estimations 

team  possibilities  considered  for  planning  of  program 

X 

no  consideration  for  outside  teaming  possibilities 

w  ork  breakdow  n  structure  (WBS)  set  up 

X 

no  WBS  completed 

Estimation/Planning  Management  (pg  2  of  2)  score  [23]  +pg  1  score  [29]  =  TOTAL  SCORE  I  52  I  Enter  on  QMM  scoresheet  blk  b. 
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Pair  choice  section  THREE:  (People  Management)  choose  most  applicable  term  of  the  two  for  each  row  (page  1  of  2): 
Human  Resources 


program  team  members  have  clearly  deined,  segmented  roles 

X 

work  responsibilities  are  shared 

formal  team  building  procedures  are  used 

X 

no  formal  team  building  emphasized 

program  manager  flexible  regarding  work  hours 

X 

program  manager  maintains  strict  standards  for  work  hours 

big  picture  conveyed  to  all  team  members  by  program  management 

X 

program  management  focuses  on  the  partitioned  tasks  with  team 

people  issues  dealt  with  primarily  through  indirect  methods  (email,  memo,  etc) 

people  issues  dealt  with  primarily  through  direct  methods  (face-to-face) 

X 

training  is  required  and  planned  on  a  regular  basis 

X 

training  is  ad  hoc 

each  team  member  is  educated  on  and  understands  overall  program  and  their  roles 

team  members  only  know  their  respective  areas 

X 

consideration  for  team  members'  career  goals  are  reflected  in  assignments 

X 

team  members  must  adapt  to  tasks  that  are  assigned 

team  members  assignments  and  responsibilities  are  mostly  dictated  by  PM 

assignments  and  responsibilities  are  discussed  and  agreed  upon  with  PM 

X 

management  leads  in  problem  solving 

management  facilitates  and  lets  team  lead  in  problem  solving 

X 

management  welcomes  problems  as  challenges  and  opportunities 

X 

management  views  problems  as  obstacles  and  grounds  for  punishment 

team  members  participate  in  performance  evaluations  of  peers 

X 

Personnel  evaluations  are  strictly  PM  responsibility 

management  reinforcement  feedback  sparse  and  inconsistent,  if  any 

management  provides  timely  reinforcement  feedback  for  positive  behaviors 

X 

management  provides  basic  needs  of  office  facilities  fairly  well 

X 

office  facilities  are  a  drawback  to  working  in  the  program 

working  conditions  are  fairly  comfortable,  time  off  policy  fairly  good 

X 

working  conditions  and  time  off  policy  is  inconsistent  and  difficult  at  times 

Communication: 


communications  primarily  written  (email) 

X 

communications  primarily  verbal  (face-to-face) 

detailed  instructions:  oral  presentation,  follow-up  email 

X 

email  only 

formal  communication  protocol 

X 

informal  communications 

external  vertical  communications  restricted 

external  vertical  communication  allowed 

X 

coders  notebook  weekly  accomplishment  reports  required 

X 

not  required 

user-coder  relationship  established,  encouraged,  and  mediated 

X 

user-coder  interaction  minimized 

meetings  structured  to  minimize  waster  time 

X 

meetings  unstructured  and  open  ended 

meetings  have  agenda,  objectives,  and  conclude  with  action  items 

X 

meeting  agenda  fluid  and  open  ended 

program  management  and  coder  communication  face  to  face 

program  management  and  coder  communication  primarily  email 

X 

program  team  updated  regularly  regarding  organizational  &  program  status 

X 

meetings  infrequently  scheduled 

open  communications  is  encouraged 

X 

communication  hrough  chain  of  command  only  is  encouraged 

program  manager  accessible  for  discussions 

X 

program  manager  difficult  to  get  an  appointment  to  see 

program  management  (PM)  is  viewed  as  separate  from  team 

X 

PM  mixes  with  team  frequently 

management  regularly  holds  team  meetings 

meetings  are  sporadic 

X 

meetings  are  structured  with  definite  goals  and  objectives 

X 

meetings  are  informal 

program  management  is  generally  easy  to  reach  and  talk  to 

X 

PM  is  usually  hard  to  get  a  hold  of  and  difficult  to  talk  to 

team-program  manager  relationship  adult-adult 

X 

team-program  management  relationship  parent-child 

schedules  are  spontaneous  and  poorly  communicated 

schedules  must  be  fixed  and  rigidly  followed  and  formally  reported 

X 

work  is  seen  as  complex  processes  involving  team  working  together 

X 

work  broken  into  pieces  with  minimal  team  member  interaction 

action  items  often  is  poorly  disseminated  and  usually  not  followed  through 

action  items  communicated  and  followed  through  thoroughly 

X 

team  members  require  frequent  clarifications  by  PM  for  assigned  tasks 

team  members  rarly  require  clarifications  by  PM  for  assigned  tasks 

X 
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Pair  choice  section  THREE:  (People  Management)  choose  most  applicable  term  of  the  two  for  each  row  (page  2  of  2): 
Leadership: 


long  range  organizational  vision 

X 

short  tern  program  and  immediate  work  focus 

lead  through  personal  attention  to  others 

action-oriented  leadership  approach 

X 

run  as  much  of  the  organization  as  possible 

let  team  make  decisions  as  much  as  possible 

X 

direct  and  domineering  style 

encourage  independence  in  others 

X 

traditional  leaders  respect  hierarchy 

do  what  needs  to  be  done 

X 

win  cooperation  rather  than  demand  it 

X 

tough-minded  with  others 

act  strongly  and  forcefully  in  the  field  of  ideas 

X 

prefer  to  lead  other  independent  types  while  seeking  autonomy  for  self 

consults  with  team  members  to  find  solutions  to  problems 

X 

consults  team  members  to  get  validation  of  PM's  predetermined  solutions 

keep  people  well  informed 

X 

only  as  much  knowledge  as  necessary  for  their  work 

make  things  happen  by  focusing  on  the  immediate  problems 

X 

long  range  focus  and  de-emphasize  current  problems 

X 

manage  others  loosely  and  prefer  minimal  supervision 

follow  traditional  procedures  and  rules  conscientiously 

leadership,  management  decisions  exclusively  by  program  management 

X 

program  management  makes  decisions  but  gets  inputs  from  team 

X 

team-program  manager  relationship  adult-adult 

team-program  management  relationship  parent-child 

program  management  makes  decisions  but  gets  inputs  from  team 

X 

all  program  team  members  responsible  for  program  decisions 

when  a  problem  arises:  management  takes  over  to  solve  it 

X 

management  lets  the  team  solve  the  problems 

leadership  is  do  as  1  say,  not  do  as  1  do 

X 

leadership  by  example 

program  expectation  not  influenced  by  PM 

X 

program  expectation  managed  by  PM 

PM  gives  freedom  to  team,  but  has  no  mentoring  for  members  (abdication) 

X 

PM  empowers  teams  by  mentoring  members  to  be  leaders 

promgram  management  waits  and  sees  what  happens  then  plans 

X 

management  plans  far  in  advance 

program  management  is  constantly  reacting  to  emergencies 

management  is  one  step  ahead  of  problems 

X 

facilitative  approach  to  solving  problems 

take  charge  readily  and  often 

X 

program  management  is  complex,  takes  much  time  to  understand 

X 

management  is  simple,  easy  to  figure  out 

X 

program  management  prefers  to  plunge  right  in 

takes  time  to  separate  things  to  be  done  and  order  of  doing  them 

X 

program  management  reacts  spur  of  the  moment 

methodically  follows  plans 

X 

Technical  Competency  of  the  Program  Manager: 

PM  has  technical  experience  particular  to  the  particular  s/w  program 
PM  participates  in  technical  reviews 

PM  participates  in  making  technical  decisions  when  problems  arise 

PM  does  not  get  involved  discussing  technical  options 

PM  does  not  review  technical  options  and  decisions 

PM  actively  attempts  to  keep  up-to-date  with  current  technology  and  standards 

PM  receives  technical  periodicals  and  occasionally  references  applicable  articles 

PM  doesn't  have  technical  background  (or  education) 

team  members  avoid  PM  when  they  need  technical  advice 


X 


_X 

X 


PM  relies  on  team  members  solely 

PM  only  in  non-technical  reviews 

PM  delegates  technical  questions 

PM  contributes  to  technical  options  being  discussed 

PM  reviews  technical  options  and  decisions 

PM  is  removed  from  cutting  edge  technology  issues 

PM  doesn't  read  periodicals  nor  reference  current  articles  to  team 

PM  has  technical  background  (or  education) 

team  members  generally  consider  talking  to  PM  regarding  technical  issues 


X 


X 

X 

X 


HR  [13]  +  Comm.  [18]  +  Leadership  [16]  +  Tech.  Competency  [7]  =  People  Mgmt.  score  [54]  Enter  on  QMM  scoresheet  blk  c. 
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Pair  choice  section  FOUR:  (Risk  Management(RM))  choose  most  applicable  term  of  the  two  for  each  row  (page  1  of  2): 


RM  is  formal  and  documented 

RM  is  informal,  if  at  all 

a  risk  management  plan  exists 

X 

no  risk  management  plan  is  developed 

RM  is  more  of  a  data  call  than  a  useful  document 

X 

RM  drives  decisions  on  the  program 

RM  is  done  prior  to  the  program  beginning 

RM  is  done  prior  and  during  program  execution 

RM  is  only  done  during  the  program  execution 

RM  is  done  prior  and  during  program  execution 

risks  are  generalized  through  the  whole  program 

risks  are  categorized 

risk  management  is  done  internally,  only 

an  outside  organization  also  contributes  to  the  RM  process 

risk  is  a  management  function 

risk  is  a  program  team  function 

risks  are  precisely  articulated 

risks  are  generalized,  if  at  all 

each  risk  has  a  consequence 

consequences  are  generalized,  if  at  all 

a  mitigation  strategy  is  completed  for  each  risk 

mitigation  strategy  is  generalized,  if  at  all 

contingency  plans  are  developed  for  a  RM  plan 

contingency  plans  are  ad  hoc  as  problems  arise  in  the  program 

risks  are  anticipated 

if  problems  arise,  management  will  deal  with  it 

the  program  doesn't  have  any  risk 

programs  that  do  not  have  risk,  have  problems 

risk  management  is  automated 

risk  management  may  use  tools,  but  depend  on  human  input 

risks  are  assigned  probabilities 

X 

probabilities  are  not  relevant  for  RM 

all  risks  are  potential  problems,  relative  priorities  for  risks  are  not  useful 

risks  are  weighed  relative  to  other  program  risks  and  thus  prioritized 

risk  management  information  is  only  shared  internally 

risk  management  information  is  shared  with  all  stakeholders 

risk  analysis  uses  ordinal  rankings 

X 

risk  analysis  uses  actual  measurements  with  a  mathematical  model 

regret  analysis  used 

no  regret  analysis  done 

attach  probabilities  to  future  events 

no  probabilities  associated  with  future  events 

assessing  risks  with  mechanical  meethods 

risks  should  be  compared  to  other  risks  and  sorted 

risk  status  tracked 

X 

not  tracked 

technical  risks  examined 

X 

no  technical  risks  examined 

process  risks  examined 

X 

no  process  risks  examined 

product  risks  examined 

X 

no  product  risks  examined 

stakeholder/user  risks  examined 

X 

no  examination  of  stakeholder/user  risks 

checklists  used  to  identify  risks 

X 

no  checklists  used 

risks  are  tracked 

X 

no  tracking  or  monitoring  of  risks 

each  risk  has  an  impact 

no  impact  analysis  of  risk 

each  risk  has  a  mitigation  plan 

X 

no  individual  risk  mitigation 

risks  monitored  by  priority 

X 

no  special  attention  to  track  higher  priority  risks 

risk  assessment  is  formalized 

no  formal  risk  assessment 

risk  control  is  formalized 

no  formal  risk  control 

integration  risks  not  considered 

integration  risks  examined 

Risk  Management  (page  1  of  2)  score  I 

22 
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Pair  choice  section  FOUR:  (Risk  Management(RM))  choose  most  applicable  term  of  the  two  for  each  row  (page  2  of  2): 


risks  to  cost 

X 

no  cost  risks  examined 

unforeseen  risks  have  occurred  in  program 

any  risk  that  came  up  had  been  identified  previously 

X 

personnel  risks  examined 

X 

no  personnel  risks  examined 

estimation  risks  examined 

X 

no  estimation  risks  examined 

planning  risks  examined 

X 

no  planning  risks  examined 

requirements  risks  examined 

X 

no  requirements  risks  examined 

resource  risks  examined 

X 

no  resource  risks  examined 

risk  management  plan  updated  regularly 

X 

no  regular  risk  management  plan  updates 

risks  charted 

risks  not  charted 

X 

performance  risks  examined 

X 

performance  risks  not  examined 

program  management  self  risks  examined 

no  program  management  risks  examined 

X 

risk  from  program  constraints  examined 

X 

no  program  constraint  risks  examined 

each  category  of  risks  are  prioritized 

X 

no  prioritization 

each  category  of  risks  are  evaluated  for  impact 

X 

no  impact  analysis  performed 

each  category  of  risks  have  control  strategy 

no  control  strategy 

X 

documentation  risks  examined 

X 

no  documentation  risks  examined 

regret  matrix  tracked 

no  regret  matrix  or  not  tracked 

X 

communication  of  risk  activities  are  facilitated 

X 

no  facilitation  or  promotion  of  communication  of  risk  activities 

taxonomy-based  questionnaire  used  to  identify  risks 

taxonomy-based  questionnaire  not  used 

X 

associated  hardware  risks  examined 

X 

no  consideration  for  hardware  risks 

integration  risks  examined 

X 

integration  risks  not  examined 

communication  risks  examined 

X 

communication  risks  not  examined 

leadership  risks  examined 

X 

leadership  risks  not  considered 

risk  avoidance  considered  for  certain  risks 

X 

risk  avoidance  not  considered  for  risks 

risk  documentation  forms  used 

no  risk  documentation  forms  used 

X 

dependency  risks  examined 

X 

no  dependency  risks  examined 

alternatives  like  risk  avoidance  considered  for  high  risk  items 

X 

no  consideration  of  risk  avoidance 

documented  risk  statements  use  a  condition-consequence  type  format 

condition-consequence  of  risk  statements  not  clearly  defined 

X 

no  assignment  of  ownership  of  risk  mitigation  action 

each  risk  mitigation  action  is  assigned  to  an  individual  for  resolution 

X 

calculation  of  risk  exposure  made  (probability  X  loss,  for  each  risk) 

no  risk  exposure  calculations 

X 

oral  communication  of  risks  only 

risks  written  in  a  way  that  communicates  nature  and  status  of  factors 

X 

triggers  used  to  quantify  risk  conditions  present 

risk  conditions  present  are  all  subjective 

X 

risk  "czar'’  in  program  for  monitoring  risks 

no  special  positions/responsibilities  for  risk  monitoring 

X 

post-program  review  completed  (scheduled)  for  unanticipated  problems  ID 

X 

no  post-program  reviews  completed  or  scheduled 

X 

no  schedule  risks  examined 

risks  to  schedule  investigated 

X 

Risk  Management  (pg  2  of  2)  score  [23]  +pg  1  score  [22]  =  TOTAL  SCORE  [55]  Enter  on  QMM  scoresheet  blk  d. 
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C.  PROGRAM  B  -  PROGRAM  MANAGER 
1.  QMM  Summary  Score  Sheet 


QMM  Scoresheet 

Part  One 

Part  Two 

Total 

Importance 

Weighted 

Category 

Score 

Score 

Score 

Coefficient 

Score 

Requirements 

Management 

a 

62 

e 

48 

110 

0.92 

= 

101.2 

Est./Planning 

Management 

D 

66 

D 

53 

119 

0.67 

= 

79.73 

People 

Management 

c 

61 

g 

43 

104 

1.86 

= 

193.44 

Risk  Management 

D 

62 

D 

54 

116 

0.55 

= 

63.8 

QMM  SCORE  438.17 


Max.  QMM  score  possible  528.00 

Min.  QMM  score  possible  -130.86 

QMM  percentage  score:  86.37% 


Objective/Subjective  view  of  the  overall  success  of  program  A  on  a  scale  of  0  to  10 
(0  being  total  failure,  10  being  perfect  program  total  success) 

Survey  Participant:  Program  Manager 

Success  Score:  8.5 
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2.  Requirements  Management  Questionnaire  Responses 


No.  Requirements  Management  Questionnaire  -  Total:  Block  e  Yes  No  N/A 


1 

PM  chose  to  have  a  formal  requirements  list 

X 

2 

Requirements  recorded  in  someway 

~1C 

3 

Written  requirements  were  part  of  some  formal  document 

~xT 

4 

Written  requirements  were  informal 

X 

5 

At  least  some  requirements  were  oral  only 

~ 

6 

All  stakeholders  were  identified 

1 r 

7 

All  stakeholders  participated  in  the  requirements  extraction 

— 

8 

Some  stakeholders  participated  in  the  requirements  extraction 

IT 

9 

Management  extracted  requirements,  no  stakeholder  involvement 

— 

10 

Management  passed  requirements  to  development  team 

~ 

11 

Stakeholders  not  involvved  in  Management  extraction,  but  approved 

~xT 

12 

Management  gets  inputs  from  stakeholders,  then  develops  requirements 

1 r 

13 

Developers  w  ork  informally  w  ith  users  to  arrive  at  requirements 

— 

14 

Same  as  13,  but  management  oversees  and  formalizes 

IT 

If  a  waterfall  or  sequential  development  strategy:  | 

15 

All  requirements  complete  before  design 

16 

Some  requirements  left  incomplete  prior  to  design 

17 

Requirements  informal  prior  to  design  effort 

18 

Requirements  serve  as  input 

19 

Length  of  time  for  requirements  w  ork  greater  than  development  w  ork 

20 

Requirements  developed  in  parallel  to  design 

OR  If  a  prototype,  throwaway,  or  other  development  strategy:  | 

15 

Learn  about  requirements  through  development  efforts 

X 

16 

No  coding  until  all  requirements  are  defined 

X 

17 

Requirements  formal  prior  to  design  effort 

~ 

18 

Requirements  serve  as  output 

X 

19 

Requirements  definition  work  in  parallel  to  development  efforts 

X 

20 

Requirements  developed  in  parallel  to  design 

X 

21 

Are  requirements  frozen  at  some  phase 

22 

Change  management  exists 

X 

23 

Change  management  is  formal 

X 

24 

Project  strategy  is  consistent  throughout  development 

X 

25 

Requirements  are  updated 

X 

26 

Configuration  Management  (CM)  exists 

X 

27 

CM  is  formal 

X 

28 

Requirements  are  testable 

X 

29 

Requirements  testing  considered/implemented  during  extraction 

X 

30 

Requirements  testing  plan  exists 

X 

31 

Requirements  testing  is  formal 

X 

32 

All  requirements  have  priorities 

X 

33 

All  requirements  must  be  implemented 

— 

34 

Requirements  are  tested 

X 

35 

All  requirements  are  equally  important 

~ 

36 

At  least  some  requirements  have  priorities 

X 

37 

All  requirements  are  traceable 

X 

38 

Traceability  not  important 

~ 

39 

Each  requirement  has  an  author 

— 

40 

Who  authored  requirement  is  not  important 

— 

41 

Initial  set  of  requirements  to  be  implemented,  no  requirements  creep 

X 

42 

Structured  and  tracked  changes  to  requirements  only 

X 

43 

Change  is  inevitable,  changes  allowed  at  all  times 

~xT 

44 

Change  is  inevitable,  but  changes  limited 

X 

45 

Requirements  control  funding 

X 

46 

Requirements  history  kept 

X 

47 

Baseline  established  for  requirements  at  some  point  prior  to  develop 

X 

TOTAL  SCORING 

m 

~e~ 

0 
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3.  Estimation/Planning  Questionnaire  Responses 


No.  Estimation/Planning  Questionnaire  -  Total:  Block  f  Yes  No  N/A 


1 

A  volume  product  metric  used  (LOC,  #  of  files,  #  of  screens,  pages  of  doc) 

X 

2 

Measure  used  for  various  product  elements  (modules,  components,  CSCI) 

X 

3 

Product  measures  made  by  phase  (amt  at  implementation,  LOC  changed  at  unit  test) 

X 

4 

Other  product  attributes  measured  (FP,  throughput,  mem  cap,  cyclomatic  complexity) 

X 

5 

Product  matrics  tracked  and  updated  hroughout  program  execution 

X 

6 

Event  count  process  metric  used  (#  defects  in  test,  reqmt  changes,  milestones  met) 

X 

7 

Time  measure  process  metric  used  (cycle  time) 

X 

8 

Process  metrics  tracked  and  updated  throughout  program  execution 

X 

9 

Program  cost  estimations  made  from  product  or  process  metrics 

X 

10 

Program  cost  extimations  tracked  and  updated  to  reflect  progress/changes 

X 

11 

Factor  analysis  performed  on  program 

X 

12 

Program's  primary  purpose,  including  major  functions  and  deliverables  known 

X 

13 

Work  breakdown  structure  developed 

X 

14 

Task  estimated  with  realistic  expectations  of  productivity  probabilities 

15 

Schedules  developed  based  on  realistic  expectations 

X 

16 

Schedules  tracked  and  updated  based  on  new  information 

X 

17 

Detailed  activity  lists  used  for  clearly  defined  completed/not  completed  tasks 

18 

Quality  assurance  plan  or  similar  to  aid  in  detecting  defects  early  in  program 

19 

COCOMO  estimates  performed 

20 

CSCI  clearly  defined  and  tasked 

X 

21 

Estimates  completed  ad  hoc 

X 

22 

Gantt  charts  used  and  updated 

X 

23 

Resource  estimations  (working  hrs,  job  categories,  task  activities)  done 

X 

24 

Earned  value  established 

X 

25 

Earned  value  tracked  throughout  program 

X 

26 

Quality  expectations  established  for  product  with  users  and  stakeholders 

X 

27 

Critical  path  for  program  tasks  developed  and  tracked 

X 

28 

Measure  of  effectiveness  (MOE)  or  Figure  of  merit  established  and  tracked 

X 

29 

Estimates  are  updated  routinely 

X 

30 

Schedules  are  updated  routinely 

X 

31 

Estimations  are  made  by  program  management  (top-down) 

X 

32 

Estimateions  are  made  by  program  team  members  (bottom-up) 

X 

33 

Automated  program  tracking  used 

X 

34 

PM  usually  thorough  in  tracking  and  reporting  schedules  and  financials 

X 

35 

WBS  developed  only  as  data  call 

X 

36 

Earned  value  used  to  track  program  progress 

X 

37 

PM  insists  on  prioritizing  work  reduction  as  schedule/funding  compromised  by  stakeholders 

X 

38 

Estimations  are  done  using  both  top  down  and  bottoms  up  approaches 

X 

39 

All  program  team  members  involved  in  planning  process 

X 

40 

Hardware  also  considered  in  estimaation  process 

X 

41 

Program  history  compiled 

X 

42 

System  upgrades  (SCR)  software  changes  requests  estimated  individually 

X 

43 

Management  duties  apart  of  each  team  member's  responsibilities 

X 

44 

PM  dictates  schedules  to  program  team 

X 

45 

Code  reviews  planned  in  schedule 

X 

46 

Defined  tangible  milestones  established  for  program  tasks 

X 

47 

Test  planning  done  at  the  start  of  the  program 

X 

48 

Estimations  are  completed  by  those  performing  the  tasks 

X 

49 

Sensitivity  analysis  performed  for  program  choices 

X 

50 

Software  deployment  planning  completed 

X 
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4.  People  Management  Questionnaire  Responses 


No.  People  Management  Questionnaire  -  Total:  Block  g  Yes  No  N/A 


1 

PM  is  accessible  in  person  by  each  team  member 

X 

2 

PM  is  accessible  via  email  (memo,  letter)  by  each  team  member 

X 

3 

PM  is  accessible  via  phone  by  each  team  member 

X 

4 

PM  not  only  considers  a  person's  suitability,  not  also  desire  to  be  on  a  team 

X 

5 

PM  consults  with  each  team  member  regarding  their  career  goals 

X 

6 

PM  regularly  holds  meetings  to  inform  team  of  program  progress 

X 

7 

PM  solicits  opinions  from  team  members  before  making  decisions 

X 

8 

PM  lets  teams  make  decisions  affecting  their  work 

X 

9 

PM  freuently  makes  decisions  without  any  consultation  with  members 

X 

10 

PM  understands  the  technology/language  of  the  program 

X 

11 

PM  is  able  to  communicate  with  other  the  technical  issues  in  the  program 

X 

12 

PM  prioritized  problems  or  conflicts  within  the  program 

X 

13 

PM  assists  team  members  in  developing/advising  of  career  path 

X 

14 

PM  empowers  program  members  to  recommend  hiring  new  team  members 

X 

15 

PM  empowers  program  members  to  recommend  firings  of  other  members 

X 

16 

PM  specifically  assigns  work  to  each  program  member 

X 

17 

PM  sets  communication  protocol 

X 

18 

PM  allows  unrestricted  communications 

X 

19 

PM  encourages  or  requires  training  for  each  individual 

X 

20 

PM  takes  control  in  difficult/roblem  areas 

X 

21 

PM  looks  ahead  to  new  programs,  new  upgrades  of  existing  program 

X 

22 

PM  maintains  regular  communications  with  all  stakeholders 

X 

23 

PM  maintains  regular  communications  with  users 

X 

24 

PM  encourages  program  team  communication  with  users 

X 

25 

PM  encourages  program  team  communication  with  stakeholders 

X 

26 

PM  facilitates  horizontal  communication  within  program 

X 

27 

PM  facilitates  communication  during  integration 

X 

28 

PM  holds  meetings  without  clear  objectives 

X 

29 

PM  must  approve  all  decisions  within  the  program 

X 

30 

PM  must  approve  all  interactions  with  stakeholders 

X 

31 

PM  must  approve  all  interactions  with  users 

X 

32 

PM  makes  all  presentations  to  stakeholders/users 

X 

33 

PM  is  considered  "flexible"  in  terms  of  program  members  personal  issues 

X 

34 

PM,  at  least  occasionally,  schedules/promotes  outside  work  team  activities 

X 

35 

PM  is  readily  willing  to  listen  to  program  prblems  and  complaints 

X 

36 

PM  takes  action  to  resolve  program  problems  and  complaints 

X 

37 

PM  is  generally  respected  by  stakeholders,  users,  and  organization 

X 

38 

PM  sometimes  fails  to  grasp  important  technical  issues  in  program 

X 

39 

PM  recruits  program  team  members  from  outside  organization 

X 

40 

PM  participates  in  technical  reviews 

X 

41 

Program  personnel  have  clearly  defined  specific  tasks 

X 

42 

Although  individual's  tasks  are  specific,  each  exposed  to  the  "bigger  picture" 

X 

43 

PM  has  clearly  defined  his/her  expectations  for  each  individual 

X 

44 

PM  delegation  of  duties  is  usually  seemless  in  execution 

X 

45 

PM  acts  as  facilitator  to  solving  personnel  conflicts 

X 

46 

PM  attempts  to  motivate  individuals  on  the  program  team 

X 

47 

PM  clearly  spearates  technical  from  managerial  roles  for  individuals 

X 

48 

PM  directs  how  he/she  expects  the  task  to  be  accomplished 

X 

49 

PM  directs  what  needs  to  be  done,  but  does  not  direct  how 

X 

50 

PM  attempts  to  spotlight  individuals  in  the  program  for  positive  exposure 

X 

TOTAL  SCORING 
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5. 


Risk  Management  Questionnaire  Responses 


No.  Risk  Management  Questionnaire  -  Total:  Block  h  Yes  No  N/A 


1 

Risk  Management  (RM)  is  specifically  an  activity  in  the  program 

X 

2 

RM  is  formal  and  documented 

X 

3 

A  specific  RM  Ian  exists 

X 

4 

RM  is  required  in  the  program,  but  not  used  during  the  program 

X 

5 

RM  is  done  prior  to  the  program  execution 

X 

6 

RM  is  done  by  an  outside  entity  to  the  development 

X 

7 

RM  is  done  internally  only 

X 

8 

RM  is  both  internally  performed  and  externally  assessed 

X 

9 

RM  planning  occurs  during  or  after  major  milestones  in  the  program 

X 

10 

Risk  Assessment  is  only  a  management  function 

X 

11 

RM  is  informal  or  non  existent 

X 

12 

There  is  a  RM  plan,  but  it  is  not  updated  or  tracked 

X 

13 

Risks  are  only  generalized 

X 

14 

Each  risk  is  delineated 

X 

15 

Each  risk  has  a  consequence 

X 

16 

Each  risk  has  a  likelihood  rating  of  some  sort 

X 

17 

Each  risk  has  a  mitigation  strategy 

X 

18 

Risk  Management  is  automated 

X 

19 

Risks  are  tracked 

X 

20 

21 

Regret  analysis  performed 

X 

22 

RM  drives  decisions  in  the  program 

X 

23 

Risks  have  probabilities 

X 

24 

Risk  Management  is  ad  hoc 

X 

25 

RM  information  is  shared  with  all  stakeholders  (as  appropriate) 

X 

26 

Risks  are  weighed  relative  to  other  program  risks 

X 

27 

Risk  Assessment  is  a  program  team  activity 

X 

28 

Risk  Assessment  done  prior  to  program  start 

X 

29 

Risk  Assessment  includes  personnal  risk 

X 

30 

RM  uses  tools,  but  depends  on  human  decisions 

X 

31 

Risk  assessment  includes  cost  risks 

X 

32 

Risk  Assessment  includes  schedule  risks 

X 

33 

Risk  Assessment  includes  technology  risks 

X 

34 

Risk  Assessment  is  briefed  organization  structure  above  program  manager 

X 

35 

Risk  Assessment  includes  requirements  risks 

X 

36 

Risk  Assessment  includes  user  risks  (too  little  involvement  of  user) 

X 

37 

Risk  Assessment  includes  documentation  risks 

X 

38 

Risk  Assessment  includes  integration  risks 

X 

39 

Risk  Assessment  includes  interface  risks  (non-standard) 

X 

40 

Risk  Assessment  includes  continuing  requirements  change  (feature  creep) 

X 

41 

Risk  Assessment  includes  dependent  projects/programs  risks 

X 

42 

Documentation  proof  exists  to  demonstrate  following  risk  management  plan 

X 

43 

High  rish  have  measured  tracking  (high  profile  status) 

X 

44 

Organizational  history  used  to  search  for  risks 

X 

45 

Other  organizational  checklists  used  for  risk  assessment 

X 

46 

Internal  organizational  checklists  used  for  risk  assessment 

X 

47 

Risk  Assessment  information  contributed  to  internal  or  other  database 

X 

48 

Risk  Assessment  includes  internal  organization  risks 

X 

49 

Risk  Assessment  includes  stakeholder  risks 

X 

50 

No  risk  management  needed;  program  is  straightforward  &  understood 

X 

TOTAL  SCORING 
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6.  Pair  Choices  Responses 

Pair  choice  section  ONE:  (Requirements  Management)  choose  most  applicable  term  of  the  two  for  each  row  (page  1  of  2): 


formal  requirement  list 

X 

informal  requirement  list 

w  ritten  requirements 

X 

oral  requirements 

requirements  informal,  but  recorded 

X 

requirements  not  recorded 

requirements  as  part  of  an  SRS  (or  other  formal  repository) 

X 

requirements  informally  recorded 

requirements  taken  as  is  from  customer 

X 

look  to  reformulate,  interview  in-depth,  or  otherwise  re-validate 

only  one  development  strategy  used 

strategies  not  consistent,  used  at  different  times 

stakeholders  as  part  of  requirements  development 

X 

stakeholders  approving  requirements  after  formulated  by  development  team 

requirements  are  testable 

X 

requirements  have  no  test  plans 

informal  test  plan  or  no  test  plan 

formal  test  plan 

test  team  involved  w  ith  requirements 

X 

no  test  team  input  or  plans  during  requirements  development 

only  a  percentage  of  requirements  present  in  baseline 

X 

baseline  must  contain  all  requirements 

requirements  documentation  has  hierarchical  structure 

X 

all  requirements  must  be  implemented 

requirements  have  listed  responsible  party 

X 

requirements  origin  not  important 

requirements  documentation  have  versions 

X 

no  requirements  history 

requirements  have  specific  attribute  values 

X 

requirements  all  rank  evenly 

funding  controls  requirements  definition 

X 

requirements  definition  controls  funding 

reqquirements  are  top  dow  n 

X 

requirements  are  bottom  up 

users/stakeholders  are  identified  and  interviewed  (market  survey) 

no  special  consideration  to  identify  users/stakeholders 

each  requirement  has  a  singular  concept 

X 

some  requirements  are  compound  statements 

requirements  definition  minimized  when  funding  short 

X 

program  scope  may  reduce,  but  requirements  definition  completed 

requirements  extraction  has  formal  process 

X 

requirements  extraction  ad  hoc 

change  procedures  formal 

X 

change  procedures  ad  hoc 

users/stakeholders  somehow  involved  in  requirements  definition 

X 

program  team  only  involved  in  requirement  definition 

management  sets  requirements  for  developers 

X 

developers  at  least  partially  involved  in  setting  requirements 

requirements  changed  at  least  once  since  baseline  established  prior  to  newversion 

X 

requirements  in  baseline  has  not  changed  prior  to  newversion  or  upgrade 

no  ranking  of  requirements 

X 

requirements  have  priorities  assigned 

use-case  diagrams  (or  other  models  or  scenario  developments) 

X 

no  models  used  for  requirements  extraction 

requirements  changes  informal 

X 

requirements  changes  formal 

plan  to  "freeze"  requirements  at  some  designated  milestone 

no  provision  for  "freezing"  requirements 

requirements  must  be  traceable 

X 

origin  of  requirements  not  important 

requirements  must  be  testable 

system  developed  must  be  testable 

test  plans  to  determine  requirements  implemented 

X 

no  test  plans  needed  for  requirements  verification 

requirements  have  priorities  in  implementation 

X 

all  requirements  must  be  implemented 

some  requirements  have  multiple  statements  or  ideas 

one  idea,  one  statement  per  requirement 

Requirements  Management  (page  1  of  2)  score  |  44 
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Pair  choice  section  ONE:  (Requirements  Management)  choose  most  applicable  term  of  the  two  for  each  row  (page  2  of  2): 


j  ANSWER  THIS  BLOCK  OF  QUESTIONS  ONLY  IF  A  SEQUENTIAL  OR  WA  TERFALL  APPROACH  IS  USED  FOR  DEVELOPMENT  (Requirements  page  2  of  2)  | 

requirements  first,  then  initial  development  work 

initial  development  work  then  requirements 

requirements  documentation  driving  development 

requirements  documentation  developed  in  parallel/after  development 

user  feedback  considered  during  development 

after  development  starts,  user  feedback  serves  as  input  to  new  work 

change  management  procedures  used  strictly 

change  management  procedures  as  guidance  only 

design  decisions  prior  to  or  in  parallel  to  requirrements  development 

design  decisions  only  after  approved  requirements  stabilized 

requirements  summarized  wht  we  have  developed 

requirements  are  the  blueprint  for  development 

length  of  time  for  requirements  work  greater  than  development  work 

length  of  time  for  requirements  work  less  than  development  work 

requirements  have  design  detail 

no  design  detail  in  requirements 

requirements  creep  to  be  avoided 

requirements  creep  o.k.,  but  need  to  be  controlled 

freeze  requirements  at  some  point 

requirements  are  fluid  throughout  development 

formal  change  procedure 

informal  change  procedure 

change  management  plan 

no  change  management  plan 

requirements  ambiguity  always  present  to  some  extent 

requirements  ambuiguity  unacceptable  at  any  level 

testing  considered  up  frornt  during  requirements  determination 

testing  considered  down  the  line  during  development 

requirements  development  team  members  different  from  implementation 

those  working  on  requirements,  work  on  implementation 

start  implementation  as  early  as  possible  to  help  define  requirements 

requirements  must  be  defined  prior  to  any  implementation  work 

j  ANSWER  THIS  BLOCK  OF  QUESTIONS  ONLY  IF  A  PROTOTYPING,  THROWAWAY,  SYNCHRONIZE  &  STABILIZE,  OR  OTHER  STRATEGY  USED  | 

develop  prototype,  then  determine  requirements 

X 

determine  requirements  prior  to  any  development  work 

requirements  testing  done  after  each  iteration 

X 

no  testing 

individual  changes  as  necessary 

X 

only  block  changes  made 

development  team  decides  on  changes  after  iteration 

users  involved  with  changes 

X 

changes  based  on  feedback  only  from  user  for  correction  of  problems 

X 

changes  to  upgrade  system  and  correct  problems 

funding  controls  changes  and  change  procedures 

X 

changes  control  funding 

requirements  documentation  finalized  prior  to  development 

requirements  fluid  throughout  development  (only  freeze  at  end) 

X 

requirements  test  plans  completed  prior  to  development 

X 

requirements  test  plans  completed  after  development 

requirements  first,  then  initial  development  work 

initial  development  work  then  requirements 

X 

use  development  effort  to  learn  more  about  requirements 

X 

define  all  requirements  prior  to  coding  anything 

requirements  ambiguity  always  present  to  some  extent 

X 

requirements  ambiguity  unacceptable  at  any  level 

requirements  have  design  detail 

X 

no  design  detail  in  requirements 

user  feedback  considered  during  development 

X 

after  development  starts,  user  feedback  serves  as  input  to  new  work 

get  something  to  users  as  soon  as  possible  for  evaluation 

X 

make  sure  it  is  complete  before  releasing 

management  dictates  requirements 

development  team  visually  represent  requirements  through  rapid  prototyping 

X 

new  requirements  allowed  after  initial  requirements  defined 

X 

new  requirements  not  allowed 

Requirements  Management  (pg  2  of  2)  score  [18]  +pg  1  score  [44]  =  TOTAL  SCORE  [62]  Enter  on  QMM  scoresheet  blk  a. 
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Pair  choice  section  TWO:  (Estimation/Planning  Management)  choose  most  applicable  term  of  the  two  for  each  row  (page  1  of  2): 


at  least  one  estimation  method  used  in  program 

X 

no  estimates 

formal  derivation  of  product  metric  for  estimation  of  size 

X 

ad  hoc  size  estimation 

ad  hoc  process  evaluation 

formal  derivation  of  at  lest  one  process  metric 

develop  w  ork  breakdow  n  structure  (WBS) 

X 

assign  work  as  needs  arise 

estimates  are  developed  to  fulfill  a  data  call  only 

use  estimates  to  plan  program 

use  estimates  to  sell  program  only 

estimates  are  useful  to  the  project  tema  for  planning  purposes 

resource  evaluations  made  for  program 

X 

no  resource  evaluation  for  planning 

use  both  bottom  up  &  top  dow  n  for  estimate,  use  one  stakeholders  like 

use  both  bottom  up  &  top  dow  n  and  evaluate  significant  differences 

estimates  made  and  not  updated 

estimates  updated  throughout  program 

resources  estimations  used  to  adjust  product  size  estimate 

X 

estimations  made  irregardless  of  resources  available 

estimations  made  to  fit  budget 

budget  made  from  estimations 

estimations  compromised  to  get  program 

rather  risk  loss  of  program  than  compromise  confident  estimations 

cycle  time  estimations 

X 

no  cycle  time  estimations 

event  count  estimations 

X 

no  event  count  estimations 

lines  of  code  (LOC)  estimation 

X 

no  LOC  estimation 

function  pont  (FP)  estimation 

X 

no  FP  estimation 

estimates  by  algorithmic  methods 

X 

estimates  by  analogy 

expert  judgement  for  estimates 

X 

ad  hoc  estimates 

estimates  by  algorithmic  methods 

X 

ad  hoc  estimates 

expert  judgement  for  estimates 

estimates  by  analogy 

ad  hoc  estimates 

estimates  by  analogy 

bottom  up  estimates 

X 

expert  judgement 

top  dow  n  estimates 

X 

expert  judgement 

ad  hoc  estimates 

any  other  estimate  process 

fuzzy  logic  estimating  method 

X 

no  formal  estimation  methodology 

WBS  development  from  estimates 

X 

WBS  development  in  parallel  or  prior  to  estimation  completion 

critical  path  of  program  determined 

X 

tasks  developed  but  no  path  is  identified 

estimators  are  program  team  members 

X 

estimators  are  outside  program  team 

management  only  on  estimations 

all  team  members  involved  in  estimation  process 

estimates  updated  at  review  s 

X 

no  updates  of  estimates 

estimates  updated  at  review  s 

estimates  constantly  updates  (in  between  reviews,  to) 

estimate  procedures  stay  the  same 

X 

estimate  procedures  change 

stakeholders  are  part  of  estimation  process 

X 

stakeholders  brief  estimations  after  completion 

estimates  are  used  beyond  initial  selling  of  program 

X 

estimates  are  one  time  events,  used  for  a  specific  purpose  once 

WBS  has  objective  measure  of  completeness 

X 

important  to  have  WBS  as  guide,  not  rigid  implementation 

Estimation/Planning  Management  (page  1  of  2)  score  1  33 

Estimation/Planning  Management  (page  1  of  2)  score 
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Pair  choice  section  TWO:  (Estimation/Planning  Management)  choose  most  applicable  term  of  the  two  for  each  row  (page  2  of  2): 


life  cycle  estimates 

X 

estimates  for  program  initiation  only 

system  upgrades  (SCR)  software  change  requests  estimated  individually 

X 

systems  upgrades  estimated  as  w  hole 

estimates  for  on-gong  resources  needed  to  maintain  s/w 

X 

estimates  for  maintenance  not  done 

informal  re-estimates  during  development 

formal  re-estimates  at  pre-defined  milestones 

formal  re-estimates  w  hen  amendment  changing  the  system  is  introduced 

X 

informal  re-estimates  w  hen  amendment  changing  the  system 

person  in-charge  of  estimation  w  alks  in  a  managers  office  to  get  an  opinion 

meeting(s)  organized  for  purpose  of  performing  cost  estimations 

factor  analysis  prior  to  commencement  of  program 

X 

none  done 

change  control  procedures  set  in  place 

X 

no  set  procedures 

elapsed  time  and  actual  w  ork  time  estimates 

X 

one  or  the  other  or  neither 

no  schedule  created 

scheudle  created 

schedule  not  updated 

schedule  updated 

schedule  follow  ed 

X 

schedule  not  followed 

tasks  identification  arises  as  program  progresses 

detailed  level  tasks  identified  prior  to  program  initiation 

scope  of  program  understood  by  all 

X 

scope  not  explicitly  defined 

quality  factors  and  criteria  identified 

X 

no  explicit  quality  factors  defined 

no  project  tracking  tools  used 

project  tracking  tools  used 

CSCIs  identified  and  tasked 

X 

CSCIs  not  explicitly  identified 

expectations  are  managed  via  estimations 

X 

estimations  are  made  to  fit  preconceived  expectations 

no  cost  schedule  developed 

cost  schedule  developed 

no  resource  schedule  developed 

resource  schedule  developed 

team  members,  management  know  at  any  time  if  in  budget  &  schedule 

X 

exact  budget  &  schedule  status  somew  hat  unclear  to  at  least  some 

individual  program  phases  are  estimated 

X 

only  top  level  program  estimated 

stakeholders/users  emphasis  understood-quick  to  field  or  all  complete 

X 

program  management  sets  delivery  tradeoffs  w  ithout  outside  input 

testing  planned  w  ith  initial  program  planning 

X 

testing  not  in  initial  planning 

documentation  not  considered  ininitial  planning 

documentation  part  of  initial  planning 

hardware  considered  in  estimations 

X 

software  only  considered 

no  formal  schedule/cost  tracking 

formal  procedures  established  for  tracking  cost  and  schedule 

earned  value  set  up 

X 

earned  value  not  used 

estimations  omit  documentation  planning 

documentation  in  estimates 

training  omitted  in  estimates 

X 

training  part  of  estimates 

earned  value  set  up,  but  not  tracked 

earned  value  tracked 

detailed  planning  done  w  ith  incomplete  set  of  requirements 

X 

detailed  planning  done  w  ith  detailed  set  of  requirements 

complete  infrastructure  support  mechanism  understood  for  estimations 

X 

no  consideration  of  infrastructure  done  for  estimations 

team  possibilities  considered  for  planning  of  program 

X 

no  consideration  for  outside  teaming  possibilities 

work  breakdown  structure  (WBS)  set  up 

X 

no  WBS  completed 

Estimation/Planning  Management  (pg  2  of  2)  score  [33]  +pg  1  score  [33]  =  TOTAL  SCORE  [66]  Enter  on  QMM  scoresheet  blk  b. 
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Pair  choice  section  THREE:  (People  Management)  choose  most  applicable  term  of  the  two  for  each  row  (page  1  of  2): 
Human  Resources 


program  team  members  have  clearly  deined,  segmented  roles 


work  responsibilities  are  shared 


formal  team  building  procedures  are  used 


no  formal  team  building  emphasized 


program  manager  flexible  regarding  work  hours 


program  manager  maintains  strict  standards  for  work  hours 


big  picture  conveyed  to  all  team  members  by  program  management 


program  management  focuses  on  the  partitioned  tasks  with  team 


people  issues  dealt  with  primarily  through  indirect  methods  (email,  memo,  etc) 


people  issues  dealt  with  primarily  through  direct  methods  (face-to-face) 


training  is  required  and  planned  on  a  regular  basis 


[training  is  ad  hoc 


each  team  member  is  educated  on  and  understands  overall  program  and  their  roles 


| team  members  only  know  their  respective  areas 


consideration  for  team  members'  career  goals  are  reflected  in  assignments 


team  members  must  adapt  to  tasks  that  are  assigned 


team  members  assignments  and  responsibilities  are  mostly  dictated  by  PM 


assignments  and  responsibilities  are  discussed  and  agreed  upon  with  PM 


management  leads  in  problem  solving 


management  facilitates  and  lets  team  lead  in  problem  solving 


management  welcomes  problems  as  challenges  and  opportunities 


management  views  problems  as  obstacles  and  grounds  for  punishment 


team  members  participate  in  performance  evaluations  of  peers 


Personnel  evaluations  are  strictly  PM  responsibility 


management  reinforcement  feedback  sparse  and  inconsistent,  if  any 


management  provides  timely  reinforcement  feedback  for  positive  behaviors 


management  provides  basic  needs  of  office  facilities  fairly  well 


office  facilities  are  a  drawback  to  working  in  the  program 


working  conditions  are  fairly  comfortable,  time  off  policy  fairly  good 


working  conditions  and  time  off  policy  is  inconsistent  and  difficult  at  times 


Communication: 


communications  primarily  written  (email) 

X 

communications  primarily  verbal  (face-to-face) 

detailed  instructions:  oral  presentation,  follow-up  email 

X 

email  only 

formal  communication  protocol 

X 

informal  communications 

external  vertical  communications  restricted 

external  vertical  communication  allowed 

coders  notebook  weekly  accomplishment  reports  required 

not  required 

user-coder  relationship  established,  encouraged,  and  mediated 

X 

user-coder  interaction  minimized 

meetings  structured  to  minimize  waster  time 

X 

meetings  unstructured  and  open  ended 

meetings  have  agenda,  objectives,  and  conclude  with  action  items 

X 

meeting  agenda  fluid  and  open  ended 

program  management  and  coder  communication  face  to  face 

X 

program  management  and  coder  communication  primarily  email 

program  team  updated  regularly  regarding  organizational  &  program  status 

X 

meetings  infrequently  scheduled 

open  communications  is  encouraged 

X 

communication  hrough  chain  of  command  only  is  encouraged 

program  manager  accessible  for  discussions 

X 

program  manager  difficult  to  get  an  appointment  to  see 

program  management  (PM)  is  viewed  as  separate  from  team 

PM  mixes  with  team  frequently 

management  regularly  holds  team  meetings 

meetings  are  sporadic 

meetings  are  structured  with  definite  goals  and  objectives 

X 

meetings  are  informal 

program  management  is  generally  easy  to  reach  and  talk  to 

X 

PM  is  usually  hard  to  get  a  hold  of  and  difficult  to  talk  to 

team-program  manager  relationship  adult-adult 

X 

team-program  management  relationship  parent-child 

schedules  are  spontaneous  and  poorly  communicated 

X 

schedules  must  be  fixed  and  rigidly  followed  and  formally  reported 

work  is  seen  as  complex  processes  involving  team  working  together 

X 

work  broken  into  pieces  with  minimal  team  member  interaction 

action  items  often  is  poorly  disseminated  and  usually  not  followed  through 

action  items  communicated  and  followed  through  thoroughly 

team  members  require  frequent  clarifications  by  PM  for  assigned  tasks 

team  members  rarly  require  clarifications  by  PM  for  assigned  tasks 

70 


XX  XX  XX  XX  XX 


Pair  choice  section  THREE:  (People  Management)  choose  most  applicable  term  of  the  two  for  each  row  (page  2  of  2): 
Leadership: 


long  range  organizational  vision 

X 

short  tern  program  and  immediate  w  ork  focus 

lead  through  personal  attention  to  others 

action-oriented  leadership  approach 

X 

run  as  much  of  the  organization  as  possible 

let  team  make  decisions  as  much  as  possible 

X 

direct  and  domineering  style 

encourage  independence  in  others 

X 

traditional  leaders  respect  hierarchy 

do  w  hat  needs  to  be  done 

X 

w  in  cooperation  rather  than  demand  it 

X 

tough-minded  with  others 

act  strongly  and  forcefully  in  the  field  of  ideas 

prefer  to  lead  other  independent  types  while  seeking  auto  no  my  for  self 

X 

consults  w  ith  team  members  to  find  solutions  to  problems 

X 

consults  team  members  to  get  validation  of  FMs  predetermined  solutions 

keep  people  well  informed 

X 

only  as  much  know  ledge  as  necessary  for  their  w  ork 

make  things  happen  by  focusing  on  the  immediate  problems 

X 

long  range  focus  and  de-emphasize  current  problems 

manage  others  loosely  and  prefer  minimal  supervision 

X 

follow  traditional  procedures  and  rules  conscientiously 

leadership,  management  decisions  exclusively  by  program  management 

program  management  makes  decisions  but  gets  inputs  from  team 

X 

team-program  manager  relationship  adult-adult 

X 

team-program  management  relationship  parent-child 

program  management  makes  decisions  but  gets  inputs  from  team 

X 

all  program  team  members  responsible  for  program  decisions 

w  hen  a  problem  arises:  management  takes  over  to  solve  it 

management  lets  the  team  solve  the  problems 

X 

leadership  is  do  as  1  say,  not  do  as  1  do 

leadership  by  example 

X 

program  expectation  not  influenced  by  PM 

program  expectation  managed  by  PM 

X 

PM  gives  freedom  to  team,  but  has  no  mentoring  for  members  (abdication) 

X 

PM  empow  ers  teams  by  mentoring  members  to  be  leaders 

promgram  management  w  aits  and  sees  w  hat  happens  then  plans 

management  plans  far  in  advance 

X 

program  management  is  constantly  reacting  to  emergencies 

management  is  one  step  ahead  of  problems 

X 

facilitative  approach  to  solving  problems 

take  charge  readily  and  often 

X 

program  management  is  complex,  takes  much  time  to  understand 

X 

management  is  simple,  easy  to  figure  out 

program  management  prefers  to  plunge  right  in 

takes  time  to  separate  things  to  be  done  and  order  of  doing  them 

X 

program  management  reacts  spur  of  the  moment 

methodically  follows  plans 

X 

Technical  Competency  of  the  Program  Manager: 


PM  has  technical  experience  particular  to  the  particular  s/w  program 

X 

PM  relies  on  team  members  solely 

PM  participates  in  technical  reviews 

X 

PM  only  in  non-technical  reviews 

PM  participates  in  making  technical  decisions  w  hen  problems  arise 

X 

PM  delegates  technical  questions 

PM  does  not  get  involved  discussing  technical  options 

PM  contributes  to  technical  options  being  discussed 

X 

PM  does  not  review  technical  options  and  decisions 

PM  reviews  technical  options  and  decisions 

X 

PM  actively  attempts  to  keep  up-to-date  with  current  technology  and  standards 

X 

PM  is  removed  from  cutting  edge  technology  issues 

PM  receives  technical  periodicals  and  occasionally  references  applicable  articles 

X 

PM  doesn't  read  periodicals  nor  reference  current  articles  to  team 

PM  doesn't  have  technical  background  (or  education) 

PM  has  technical  background  (or  education) 

X 

team  members  avoid  PM  w  hen  they  need  technical  advice 

team  members  generally  consider  talking  to  PM  regarding  technical  issues 

X 

HR  [13]  +  Comm.  [18]  +  Leadership  [21]  +  Tech.  Competency  [9]  =  People  Mgmt.  score  [61]  Enter  on  QMM  scoresheet  blk  c. 
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Pair  choice  section  FOUR:  (Risk  Management(RM))  choose  most  applicable  term  of  the  two  for  each  row  (page  1  of  2): 


RM  is  formal  and  documented 

X 

RM  is  informal,  if  at  all 

a  risk  management  plan  exists 

X 

no  risk  management  plan  is  developed 

RM  is  more  of  a  data  call  than  a  useful  document 

RM  drives  decisions  on  the  program 

RM  is  done  prior  to  the  program  beginning 

RM  is  done  prior  and  during  program  execution 

RM  is  only  done  during  the  program  execution 

RM  is  done  prior  and  during  program  execution 

risks  are  generalized  through  the  whole  program 

risks  are  categorized 

risk  management  is  done  internally,  only 

an  outside  organization  also  contributes  to  the  RM  process 

risk  is  a  management  function 

risk  is  a  program  team  function 

risks  are  precisely  articulated 

X 

risks  are  generalized,  if  at  all 

each  risk  has  a  consequence 

X 

consequences  are  generalized,  if  at  all 

a  mitigation  strategy  is  completed  for  each  risk 

X 

mitigation  strategy  is  generalized,  if  at  all 

contingency  plans  are  developed  for  a  RM  plan 

X 

contingency  plans  are  ad  hoc  as  problems  arise  in  the  program 

risks  are  anticipated 

if  problems  arise,  management  w  ill  deal  w  ith  it 

the  program  doesn't  have  any  risk 

programs  that  do  not  have  risk,  have  problems 

risk  management  is  automated 

X 

risk  management  may  use  tools,  but  depend  on  human  input 

risks  are  assigned  probabilities 

X 

probabilities  are  not  relevant  for  RM 

all  risks  are  potential  problems,  relative  priorities  for  risks  are  not  useful 

risks  are  weighed  relative  to  other  program  risks  and  thus  prioritized 

risk  management  information  is  only  shared  internally 

risk  management  information  is  shared  with  all  stakeholders 

risk  analysis  uses  ordinal  rankings 

risk  analysis  uses  actual  measurements  w  ith  a  mathematical  model 

regret  analysis  used 

X 

no  regret  analysis  done 

attach  probabilities  to  future  events 

X 

no  probabilities  associated  w  ith  future  events 

assessing  risks  with  mechanical  meethods 

risks  should  be  compared  to  other  risks  and  sorted 

risk  status  tracked 

X 

not  tracked 

technical  risks  examined 

X 

no  technical  risks  examined 

process  risks  examined 

X 

no  process  risks  examined 

product  risks  examined 

X 

no  product  risks  examined 

stakeholder/user  risks  examined 

X 

no  examination  of  stakeholder/user  risks 

checklists  used  to  identify  risks 

X 

no  checklists  used 

risks  are  tracked 

X 

no  tracking  or  monitoring  of  risks 

each  risk  has  an  impact 

X 

no  impact  analysis  of  risk 

each  risk  has  a  mitigation  plan 

X 

no  individual  risk  mitigation 

risks  monitored  by  priority 

X 

no  special  attention  to  track  higher  priority  risks 

risk  assessment  is  formalized 

no  formal  risk  assessment 

risk  control  is  formalized 

no  formal  risk  control 

integration  risks  not  considered 

integration  risks  examined 

Risk  Management  (page  1  of  2)  score 

33 
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Pair  choice  section  FOUR:  (Risk  Management(RM))  choose  most  applicable  term  of  the  two  for  each  row  (page  2  of  2): 


risks  to  cost 

X 

no  cost  risks  examined 

unforeseen  risks  have  occurred  in  program 

X 

any  risk  that  came  up  had  been  identified  previously 

personnel  risks  examined 

X 

no  personnel  risks  examined 

estimation  risks  examined 

X 

no  estimation  risks  examined 

planning  risks  examined 

X 

no  planning  risks  examined 

requirements  risks  examined 

X 

no  requirements  risks  examined 

resource  risks  examined 

X 

no  resource  risks  examined 

risk  management  plan  updated  regularly 

X 

no  regular  risk  management  plan  updates 

risks  charted 

X 

risks  not  charted 

performance  risks  examined 

X 

performance  risks  not  examined 

program  management  self  risks  examined 

X 

no  program  management  risks  examined 

risk  from  program  constraints  examined 

X 

no  program  constraint  risks  examined 

each  category  of  risks  are  prioritized 

X 

no  prioritization 

each  category  of  risks  are  evaluated  for  impact 

X 

no  impact  analysis  performed 

each  category  of  risks  have  control  strategy 

X 

no  control  strategy 

documentation  risks  examined 

X 

no  documentation  risks  examined 

regret  matrix  tracked 

X 

no  regret  matrix  or  not  tracked 

communication  of  risk  activities  are  facilitated 

X 

no  facilitation  or  promotion  of  communication  of  risk  activities 

taxonomy-based  questionnaire  used  to  identify  risks 

X 

taxonomy-based  questionnaire  not  used 

associated  hardware  risks  examined 

X 

no  consideration  for  hardware  risks 

integration  risks  examined 

X 

integration  risks  not  examined 

communication  risks  examined 

X 

communication  risks  not  examined 

leadership  risks  examined 

leadership  risks  not  considered 

X 

risk  avoidance  considered  for  certain  risks 

X 

risk  avoidance  not  considered  for  risks 

risk  documentation  forms  used 

X 

no  risk  documentation  forms  used 

dependency  risks  examined 

X 

no  dependency  risks  examined 

alternatives  like  risk  avoidance  considered  for  high  risk  items 

X 

no  consideration  of  risk  avoidance 

documented  risk  statements  use  a  condition-consequence  type  format 

X 

condition-consequence  of  risk  statements  not  clearly  defined 

no  assignment  of  ownership  of  risk  mitigation  action 

X 

each  risk  mitigation  action  is  assigned  to  an  individual  for  resolution 

calculation  of  risk  exposure  made  (probability  X  loss,  for  each  risk) 

X 

no  risk  exposure  calculations 

oral  communication  of  risks  only 

risks  written  in  a  way  that  communicates  nature  and  status  of  factors 

X 

triggers  used  to  quantify  risk  conditions  present 

risk  conditions  present  are  all  subjective 

X 

risk  "czar"  in  program  for  monitoring  risks 

no  special  positions/responsibilities  for  risk  monitoring 

X 

post-program  review  completed  (scheduled)  for  unanticipated  problems  ID 

X 

no  post-program  reviews  completed  or  scheduled 

no  schedule  risks  examined 

risks  to  schedule  investigated 

X 

Risk  Management  (pg  2  of  2)  score  [29]  +pg  1  score  [33]  =  TOTAL  SCORE  [62]  Enter  on  QMM  scoresheet  blk  d. 
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D.  PROGRAM  B  -  ASSOCIATE 

1.  QMM  Summary  Score  Sheet 


QMM  Scoresheet 

Part  One 

Part  Two 

Total 

Importance 

Weighted 

Category 

Score 

Score 

Score 

Coefficient 

Score 

Requirements 

Management 

a 

60 

e 

47 

107 

0.92 

= 

98.44 

Est./Planning 

Management 

D 

64 

D 

52 

116 

0.67 

= 

77.72 

People 

Management 

c 

60 

g 

42 

102 

1.86 

= 

189.72 

Risk  Management 

D 

61 

D 

53 

114 

0.55 

= 

62.7 

QMM  SCORE  428.58 


Max.  QMM  score  possible  528.00 

Min.  QMM  score  possible  -130.86 

QMM  percentage  score:  84.91% 


Objective/Subjective  view  of  the  overall  success  of  program  B  on  a  scale  of  0  to  10 
(0  being  total  failure,  10  being  perfect  program  total  success) 

Survey  Participant:  Associate 
Success  Score:  8.5 
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2.  Requirements  Management  Questionnaire  Responses 


No.  Requirements  Management  Questionnaire  -  Total:  Block  e  Yes  No  N/A 


1 

PM  chose  to  have  a  formal  requirements  list 

X 

2 

Requirements  recorded  in  some  way 

X 

3 

Written  requirements  were  part  of  some  formal  document 

X 

4 

Written  requirements  were  informal 

X 

5 

At  least  some  requirements  were  oral  only 

X 

6 

All  stakeholders  were  identified 

X 

7 

All  stakeholders  participated  in  the  requirements  extraction 

X 

8 

Some  stakeholders  participated  in  the  requirements  extraction 

X 

9 

Management  extracted  requirements,  no  stakeholder  involvement 

— 

10 

Management  passed  requirements  to  development  team 

X 

11 

Stakeholders  not  involvved  in  Management  extraction,  but  approved 

— 

12 

Management  gets  inputs  from  stakeholders,  then  develops  requirements 

X 

13 

Developers  work  inf ormally  w ith  users  to  arrive  at  requirements 

X 

14 

Same  as  13,  but  management  oversees  and  formalizes 

X 

If  a  waterfall  or  sequential  development  strategy:  j 

15 

All  requirements  complete  before  design 

16 

Some  requirements  left  incomplete  prior  to  design 

17 

Requirements  informal  prior  to  design  effort 

18 

Requirements  serve  as  input 

19 

Length  of  time  for  requirements  work  greater  than  development  w ork 

20 

Requirements  developed  in  parallel  to  design 

OR  If  a  prototype ,  throwaway ,  or  other  development  strategy:  [ 

15 

Learn  about  requirements  through  development  efforts 

X 

16 

No  coding  until  all  requirements  are  defined 

X 

17 

Requirements  formal  prior  to  design  effort 

X 

18 

Requirements  serve  as  output 

19 

Requirements  definition  work  in  parallel  to  development  efforts 

~X~ 

20 

Requirements  developed  in  parallel  to  design 

~X~ 

21 

Are  requirements  frozen  at  some  phase 

T" 

22 

Change  management  exists 

“>T 

23 

Change  management  is  formal 

~xT 

24 

Project  strategy  is  consistent  throughout  development 

“x- 

25 

Requirements  are  updated 

“x- 

26 

Configuration  Management  (CM)  exists 

~>r 

27 

CM  is  formal 

~>r 

28 

Requirements  are  testable 

~x~ 

29 

Requirements  testing  considered/implemented  during  extraction 

~>r 

30 

Requirements  testing  plan  exists 

“x- 

31 

Requirements  testing  is  formal 

~>r 

32 

All  requirements  have  priorities 

~x~ 

33 

All  requirements  must  be  implemented 

IT 

34 

Requirements  are  tested 

~>r 

35 

All  requirements  are  equally  important 

“jT 

36 

At  least  some  requirements  have  priorities 

“x- 

37 

All  requirements  are  traceable 

TT 

38 

Traceability  not  important 

~x~ 

39 

Each  requirement  has  an  author 

“x- 

40 

Who  authored  requirement  is  not  important 

~ 

41 

Initial  set  of  requirements  to  be  implemented,  no  requirements  creep 

“x- 

42 

Structured  and  tracked  changes  to  requirements  only 

~ir 

43 

Change  is  inevitable,  changes  allowed  at  all  times 

~ 

44 

Change  is  inevitable,  but  changes  limited 

45 

Requirements  control  funding 

~ir 

46 

Requirements  history  kept 

— 

47 

Baseline  established  for  requirements  at  some  point  prior  to  develop 

— 

TOTAL  SCORING 

m 
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3.  Estimation/Planning  Questionnaire  Responses 


No.  Estimation/Planning  Questionnaire  -  Total:  Block  f  Yes  No  N/A 


1 

Avolume  product  metric  used  (LOC,  #  of  files,  #  of  screens,  pages  of  doc) 

X 

Total 

2 

Measure  used  for  various  product  elements  (modules,  components,  CSCI) 

X 

3 

Product  measures  made  by  phase  (amt  at  implementation,  LOC  changed  at  unit  test) 

X 

4 

Other  product  attributes  measured  (FP,  throughput,  mem  cap,  cyclomatic  complexity) 

X 

5 

Product  matrics  tracked  and  updated  hroughout  program  execution 

X 

6 

Event  count  process  metric  used  (#  defects  in  test,  reqmt  changes,  milestones  met) 

X 

7 

Time  measure  process  metric  used  (cycle  time) 

X 

8 

Process  metrics  tracked  and  updated  throughout  program  execution 

X 

9 

Program  cost  estimations  made  from  product  or  process  metrics 

X 

10 

Program  costextimations  tracked  and  updated  to  reflect  progress/changes 

X 

11 

Factor  analysis  performed  on  program 

X 

12 

Program's  primary  purpose,  including  major  functions  and  deliverables  known 

X 

13 

Work  breakdown  structure  developed 

X 

14 

Task  estimated  with  realistic  expectations  of  productivity  probabilities 

X 

15 

Schedules  developed  based  on  realistic  expectations 

X 

16 

Schedules  tracked  and  updated  based  on  new  information 

X 

17 

Detailed  activity  lists  used  for  clearly  defined  completed/not  completed  tasks 

X 

18 

Quality  assurance  plan  orsimilarto  aid  in  detecting  defects  earlyin  program 

X 

19 

COCOMO  estimates  performed 

X 

20 

CSCI  clearly  defined  and  tasked 

X 

21 

Estimates  completed  ad  hoc 

X 

22 

Gantt  charts  used  and  updated 

X 

23 

Resource  estimations  (working  hrs,  job  categories,  task  activities)  done 

X 

24 

Earned  value  established 

X 

25 

Earned  value  tracked  throughout  program 

X 

26 

Quality  expectations  established  for  product  with  users  and  stakeholders 

X 

27 

Critical  path  for  program  tasks  developed  and  tracked 

X 

28 

Measure  of  effectiveness  (MOE)  or  Figure  of  merit  established  and  tracked 

X 

29 

Estimates  are  updated  routinely 

X 

30 

Schedules  are  updated  routinely 

X 

31 

Estimations  are  made  by  program  management  (top-down) 

X 

32 

Estimateions  are  made  by  program  team  members  (bottom-up) 

X 

33 

Automated  program  tracking  used 

X 

34 

PM  usually  thorough  in  tracking  and  reporting  schedules  and  financials 

X 

35 

WBS  developed  only  as  data  call 

X 

36 

Earned  value  used  to  track  program  progress 

X 

37 

PM  insists  on  prioritizing  work  reduction  as  schedule/funding  compromised  by  stakeholders 

X 

38 

Estimations  are  done  using  both  top  down  and  bottoms  up  approaches 

X 

39 

All  program  team  members  involved  in  planning  process 

X 

40 

Hardware  also  considered  in  estimaation  process 

X 

41 

Program  history  compiled 

X 

42 

System  upgrades  (SCR)  software  changes  requests  estimated  individually 

X 

43 

Management  duties  apart  of  each  team  member's  responsibilities 

X 

44 

PM  dictates  schedules  to  program  team 

X 

45 

Code  reviews  planned  in  schedule 

X 

46 

Defined  tangible  milestones  established  for  program  tasks 

X 

47 

Test  planning  done  at  the  start  of  the  program 

X 

48 

Estimations  are  completed  bythose  performing  the  tasks 

X 

49 

Sensitivity  analysis  performed  for  program  choices 

X 

50 

Software  deployment  planning  completed 

X 

TOTAL  SCORING 

m 

1 

_“l 
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4.  People  Management  Questionnaire  Responses 


No.  People  Management  Questionnaire  -  Total:  Block  g  Yes  No  N/A 


1 

PM  is  accessible  in  person  by  each  team  member 

X 

2 

PM  is  accessible  via  email  (memo,  letter)  by  each  team  member 

X 

3 

PM  is  accessible  via  phone  by  each  team  member 

X 

4 

PM  not  only  considers  a  person's  suitability,  not  also  desire  to  be  on  a  team 

X 

5 

PM  consults  with  each  team  member  regarding  their  career  goals 

X 

6 

PM  regularly  holds  meetings  to  inform  team  of  program  progress 

X 

7 

PM  solicits  opinions  from  team  members  before  making  decisions 

X 

8 

PM  lets  teams  make  decisions  affecting  their  work 

X 

9 

PM  freuently  makes  decisions  without  any  consultation  with  members 

X 

10 

PM  understands  the  technology/language  of  the  program 

X 

11 

PM  is  able  to  communicate  with  other  the  technical  issues  in  the  program 

X 

12 

PM  prioritized  problems  or  conflicts  within  the  program 

X 

13 

PM  assists  team  members  in  developing/advising  of  career  path 

X 

14 

PM  empowers  program  members  to  recommend  hiring  new  team  members 

X 

15 

PM  empowers  program  members  to  recommend  firings  of  other  members 

X 

16 

PM  specifically  assigns  work  to  each  program  member 

X 

17 

PM  sets  communication  protocol 

X 

18 

PM  allows  unrestricted  communications 

X 

19 

PM  encourages  or  requires  training  for  each  individual 

X 

20 

PM  takes  control  in  difficult/roblem  areas 

X 

21 

PM  looks  ahead  to  new  programs,  new  upgrades  of  existing  program 

X 

22 

PM  maintains  regular  communications  with  all  stakeholders 

X 

23 

PM  maintains  regular  communications  with  users 

X 

24 

PM  encourages  program  team  communication  with  users 

X 

25 

PM  encourages  program  team  communication  with  stakeholders 

X 

26 

PM  facilitates  horizontal  communication  within  program 

X 

27 

PM  facilitates  communication  during  integration 

X 

28 

PM  holds  meetings  without  clear  objectives 

X 

29 

PM  must  approve  all  decisions  within  the  program 

X 

30 

PM  must  approve  all  interactions  with  stakeholders 

X 

31 

PM  must  approve  all  interactions  with  users 

X 

32 

PM  makes  all  presentations  to  stakeholders/users 

X 

33 

PM  is  considered  "flexible"  in  terms  of  program  members  personal  issues 

X 

34 

PM,  at  least  occasionally,  schedules/promotes  outside  work  team  activities 

X 

35 

PM  is  readily  willing  to  listen  to  program  prblems  and  complaints 

X 

36 

PM  takes  action  to  resolve  program  problems  and  complaints 

X 

37 

PM  is  generally  respected  by  stakeholders,  users,  and  organization 

X 

38 

PM  sometimes  fails  to  grasp  important  technical  issues  in  program 

X 

39 

PM  recruits  program  team  members  from  outside  organization 

X 

40 

PM  participates  in  technical  reviews 

X 

41 

Program  personnel  have  clearly  defined  specific  tasks 

X 

42 

Although  individual's  tasks  are  specific,  each  exposed  to  the  "bigger  picture" 

X 

43 

PM  has  clearly  defined  his/her  expectations  for  each  individual 

X 

44 

PM  delegation  of  duties  is  usually  seemless  in  execution 

X 

45 

PM  acts  as  facilitator  to  solving  personnel  conflicts 

X 

46 

PM  attempts  to  motivate  individuals  on  the  program  team 

X 

47 

PM  clearly  spearates  technical  from  managerial  roles  for  individuals 

X 

48 

PM  directs  how  he/she  expects  the  task  to  be  accomplished 

X 

49 

PM  directs  what  needs  to  be  done,  but  does  not  direct  how 

X 

50 

PM  attempts  to  spotlight  individuals  in  the  program  for  positive  exposure 

X 

X 

TOTAL  SCORING 
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5. 


Risk  Management  Questionnaire  Responses 


No.  Risk  Management  Questionnaire  -  Total:  Block  h  Yes  No  N/A 


1 

Risk  Management  (RM)  is  specifically  an  activity  in  the  program 

X 

2 

RM  is  formal  and  documented 

X 

3 

A  specific  RM  Ian  exists 

X 

4 

RM  is  required  in  the  program,  but  not  used  during  the  program 

X 

5 

RM  is  done  prior  to  the  program  execution 

X 

6 

RM  is  done  by  an  outside  entity  to  the  development 

X 

7 

RM  is  done  internally  only 

X 

8 

RM  is  both  internally  performed  and  externally  assessed 

X 

9 

RM  planning  occurs  during  or  after  major  milestones  in  the  program 

X 

10 

Risk  Assessment  is  only  a  management  function 

X 

11 

RM  is  informal  or  non  existent 

X 

12 

There  is  a  RM  plan,  but  it  is  not  updated  or  tracked 

X 

13 

Risks  are  only  generalized 

X 

14 

Each  risk  is  delineated 

X 

15 

Each  risk  has  a  consequence 

X 

16 

Each  risk  has  a  likelihood  rating  of  some  sort 

X 

17 

Each  risk  has  a  mitigation  strategy 

X 

18 

Risk  Management  is  automated 

X 

19 

Risks  are  tracked 

X 

20 

21 

Regret  analysis  performed 

X 

22 

RM  drives  decisions  in  the  program 

X 

23 

Risks  have  probabilities 

X 

24 

Risk  Management  is  ad  hoc 

X 

25 

RM  information  is  shared  with  all  stakeholders  (as  appropriate) 

X 

26 

Risks  are  weighed  relative  to  other  program  risks 

X 

27 

Risk  Assessment  is  a  program  team  activity 

X 

28 

Risk  Assessment  done  prior  to  program  start 

X 

29 

Risk  Assessment  includes  personnal  risk 

X 

30 

RM  uses  tools,  but  depends  on  human  decisions 

X 

31 

Risk  assessment  includes  cost  risks 

X 

32 

Risk  Assessment  includes  schedule  risks 

X 

33 

Risk  Assessment  includes  technology  risks 

X 

34 

Risk  Assessment  is  briefed  organization  structure  above  program  manager 

X 

35 

Risk  Assessment  includes  requirements  risks 

X 

36 

Risk  Assessment  includes  user  risks  (too  little  involvement  of  user) 

X 

37 

Risk  Assessment  includes  documentation  risks 

X 

38 

Risk  Assessment  includes  integration  risks 

X 

39 

Risk  Assessment  includes  interface  risks  (non-standard) 

X 

40 

Risk  Assessment  includes  continuing  requirements  change  (feature  creep) 

X 

41 

Risk  Assessment  includes  dependent  projects/programs  risks 

X 

42 

Documentation  proof  exists  to  demonstrate  following  risk  management  plan 

X 

43 

High  rish  have  measured  tracking  (high  profile  status) 

X 

44 

Organizational  history  used  to  search  for  risks 

X 

45 

Other  organizational  checklists  used  for  risk  assessment 

X 

46 

Internal  organizational  checklists  used  for  risk  assessment 

X 

47 

Risk  Assessment  information  contributed  to  internal  or  other  database 

X 

48 

Risk  Assessment  includes  internal  organization  risks 

X 

49 

Risk  Assessment  includes  stakeholder  risks 

X 

50 

No  risk  management  needed;  program  is  straightforward  &  understood 

X 

TOTAL  SCORING 
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6.  Pair  Choices  Responses 

Pair  choice  section  ONE:  (Requirements  Management)  choose  most  applicable  term  of  the  two  for  each  row  (page  1  of  2): 


formal  requirement  list 

X 

informal  requirement  list 

written  requirements 

X 

oral  requirements 

requirements  informal,  but  recorded 

mm 

requirements  not  recorded 

requirements  as  part  of  an  SRS  (or  other  formal  repository) 

X 

requirements  informally  recorded 

requirements  taken  as  is  from  customer 

look  to  reformulate,  interview  in-depth,  or  otherwise  re-validate 

only  one  development  strategy  used 

mm 

strategies  not  consistent,  used  at  different  times 

stakeholders  as  part  of  requirements  development 

mm 

stakeholders  approving  requirements  after  formulated  by  development  team 

requirements  are  testable 

X 

requirements  have  no  test  plans 

informal  test  plan  or  no  test  plan 

formal  test  plan 

test  team  involved  with  requirements 

mm 

no  test  team  input  or  plans  during  requirements  development 

only  a  percentage  of  requirements  present  in  baseline 

baseline  must  contain  all  requirements 

requirements  documentation  has  hierarchical  structure 

X 

all  requirements  must  be  implemented 

requirements  have  listed  responsible  party 

mm 

requirements  origin  not  important 

requirements  documentation  have  versions 

1  X  1 

no  requirements  history 

requirements  have  specific  attribute  values 

mm 

requirements  all  rank  evenly 

funding  controls  requirements  definition 

requirements  definition  controls  funding 

reqquirements  are  top  down 

mm 

requirements  are  bottom  up 

users/stakeholders  are  identified  and  interviewed  (market  survey) 

1  X  1 

no  special  consideration  to  identify  users/stakeholders 

each  requirement  has  a  singular  concept 

ma 

some  requirements  are  compound  statements 

requirements  definition  minimized  when  funding  short 

mm 

program  scope  may  reduce,  but  requirements  definition  completed 

requirements  extraction  has  formal  process 

1  X  1 

requirements  extraction  ad  hoc 

change  procedures  formal 

ma 

change  procedures  ad  hoc 

users/stakeholders  somehow  involved  in  requirements  definition 

mm 

program  team  only  involved  in  requirement  definition 

management  sets  requirements  for  developers 

developers  at  least  partially  involved  in  setting  requirements 

requirements  changed  at  least  once  since  baseline  established  prior  to  new  version 

requirements  in  baseline  has  not  changed  prior  to  new  version  or  upgrade 

no  ranking  of  requirements 

requirements  have  priorities  assigned 

use-case  diagrams  (or  other  models  or  scenario  developments) 

mm 

no  models  used  for  requirements  extraction 

requirements  changes  informal 

requirements  changes  formal 

plan  to  "freeze"  requirements  at  some  designated  milestone 

no  provision  for  "freezing"  requirements 

requirements  must  be  traceable 

origin  of  requirements  not  important 

requirements  must  be  testable 

system  developed  must  be  testable 

test  plans  to  determine  requirements  implemented 

no  test  plans  needed  for  requirements  verification 

requirements  have  priorities  in  implementation 

all  requirements  must  be  implemented 

some  requirements  have  multiple  statements  or  ideas 

one  idea,  one  statement  per  requirement 

Requirements  Management  (page  1  of  2)  score 
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Pair  choice  section  ONE:  (Requirements  Management)  choose  most  applicable  term  of  the  two  for  each  row  (page  2  of  2): 


|  AN  S  WE  R  THIS  BLOCK  OF  QUESTIONS  ONLY  IF  A  SEQUENTIAL  OR  WAT  ERF  ALL  AP  P  ROACH  IS  USED  FOR  DEVELOP  M  ENT  (Requirements  page  2  of  2)  j 

requirements  first,  then  initial  development  w  ork 

initial  development  w  ork  then  requirements 

requirements  documentation  driving  development 

requirements  documentation  developed  in  parallel/after  development 

user  feedback  considered  during  development 

after  development  starts,  user  feedback  serves  as  input  to  new  work 

change  management  procedures  used  strictly 

change  management  procedures  as  guidance  only 

design  decisions  prior  to  or  in  parallel  to  requirrements  development 

design  decisions  only  after  approved  requirements  stabilized 

requirements  summarized  w  ht  w  e  have  developed 

requirements  are  the  blueprint  for  development 

length  of  time  for  requirements  w  ork  greater  than  development  w  ork 

length  of  time  for  requirements  work  less  than  development  w ork 

requirements  have  design  detail 

no  design  detail  in  requirements 

requirements  creep  to  be  avoided 

requirements  creep  o.k.,  but  need  to  be  controlled 

freeze  requirements  at  some  point 

requirements  are  fluid  throughout  development 

formal  change  procedure 

informal  change  procedure 

change  management  plan 

no  change  management  plan 

requirements  ambiguity  always  present  to  some  extent 

requirements  ambuiguity  unacceptable  at  any  level 

testing  considered  up  frornt  during  requirements  determination 

testing  considered  down  the  line  during  development 

requirements  development  team  members  different  from  implementation 

those  working  on  requirements,  work  on  implementation 

start  implementation  as  early  as  possible  to  help  define  requirements 

requirements  must  be  defined  prior  to  any  implementation  work 

ANSWER  THIS  BLOCK  OF  QUESTIONS  ONLY  IF  A  PROTOTYPING,  THROWAWAY,  SYNCHRONIZE  &  STABILIZE,  OR  OTHER  STRATEGY  USED 

develop  prototype,  then  determine  requirements 

X 

determine  requirements  prior  to  any  development  work 

requirements  testing  done  after  each  iteration 

no  testing 

X 

individual  changes  as  necessary 

X 

only  block  changes  made 

X 

development  team  decides  on  changes  after  iteration 

X 

users  involved  w  ith  changes 

changes  based  on  feedback  only  from  user  for  correction  of  problems 

changes  to  upgrade  system  and  correct  problems 

X 

funding  controls  changes  and  change  procedures 

X 

changes  control  funding 

requirements  documentation  finalized  prior  to  development 

requirements  fluid  throughout  development  (only freeze  at  end) 

X 

requirements  test  plans  completed  prior  to  development 

X 

requirements  test  plans  completed  after  development 

requirements  first,  then  initial  development  w  ork 

X 

initial  development  work  then  requirements 

use  development  effort  to  learn  more  about  requirements 

X 

define  all  requirements  prior  to  coding  anything 

requirements  ambiguity  always  present  to  some  extent 

X 

requirements  ambiguity  unacceptable  at  any  level 

requirements  have  design  detail 

no  design  detail  in  requirements 

X 

user  feedback  considered  during  development 

after  development  starts,  user  feedback  serves  as  input  to  new  work 

X 

get  something  to  users  as  soon  as  possible  for  evaluation 

X 

make  sure  it  is  complete  before  releasing 

management  dictates  requirements 

X 

development  team  visually  represent  requirements  through  rapid  prototyping 

new  requirements  allowed  after  initial  requirements  defined 

X 

new  requirements  not  allow  ed 

Requirements  Management  (pg  2  of  2)  score  [1 5]  +pg  1  score  [45]  =  TOTAL  SCORE  [60]  Enter  on  QMM  scoresheet  blk  a. 
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Pair  choice  section  TWO:  (Estimation/Planning  Management)  choose  most  applicable  term  of  the  two  for  each  row  (page  1  of  2): 


at  least  one  estimation  method  used  in  program 

X 

no  estimates 

formal  derivation  of  product  metric  for  estimation  of  size 

X 

ad  hoc  size  estimation 

ad  hoc  process  evaluation 

formal  derivation  of  at  lest  one  process  metric 

develop  work  breakdown  structure  (WBS) 

X 

assign  work  as  needs  arise 

estimates  are  developed  to  fulfill  a  data  call  only 

use  estimates  to  plan  program 

use  estimates  to  sell  program  only 

estimates  are  useful  to  the  project  tema  for  planning  purposes 

resource  evaluations  made  for  program 

X 

no  resource  evaluation  for  planning 

use  both  bottom  up  &  top  down  for  estimate,  use  one  stakeholders  like 

use  both  bottom  up  &  top  down  and  evaluate  significant  differences 

estimates  made  and  not  updated 

estimates  updated  throughout  program 

resources  estimations  used  to  adjust  product  size  estimate 

X 

estimations  made  irregardless  of  resources  available 

estimations  made  to  fit  budget 

budget  made  from  estimations 

estimations  compromised  to  get  program 

rather  risk  loss  of  program  than  compromise  confident  estimations 

cycle  time  estimations 

X 

no  cycle  time  estimations 

event  count  estimations 

X 

no  event  count  estimations 

lines  of  code  (LOC)  estimation 

X 

no  LOC  estimation 

function  pont  (FP)  estimation 

X 

no  FP  estimation 

estimates  by  algorithmic  methods 

X 

estimates  by  analogy 

expert  judgement  for  estimates 

X 

ad  hoc  estimates 

estimates  by  algorithmic  methods 

X 

ad  hoc  estimates 

expert  judgement  for  estimates 

estimates  by  analogy 

ad  hoc  estimates 

estimates  by  analogy 

bottom  up  estimates 

X 

expert  judgement 

top  down  estimates 

X 

expert  judgement 

ad  hoc  estimates 

any  other  estimate  process 

fuzzy  logic  estimating  method 

X 

no  formal  estimation  methodology 

WBS  development  from  estimates 

X 

WBS  development  in  parallel  or  prior  to  estimation  completion 

critical  path  of  program  determined 

X 

tasks  developed  but  no  path  is  identified 

estimators  are  program  team  members 

X 

estimators  are  outside  program  team 

management  only  on  estimations 

all  team  members  involved  in  estimation  process 

estimates  updated  at  reviews 

X 

no  updates  of  estimates 

estimates  updated  at  reviews 

estimates  constantly  updates  (in  between  reviews,  to) 

estimate  procedures  stay  the  same 

X 

estimate  procedures  change 

stakeholders  are  part  of  estimation  process 

X 

stakeholders  brief  estimations  after  completion 

estimates  are  used  beyond  initial  selling  of  program 

X 

estimates  are  one  time  events,  used  for  a  specific  purpose  once 

WBS  has  objective  measure  of  completeness 

X 

important  to  have  WBS  as  guide,  not  rigid  implementation 

Estimation/Planning  Management  (page  1  of  2)  score 
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Pair  choice  section  TWO:  (Estimation/Planning  Management)  choose  most  applicable  term  of  the  two  for  each  row  (page  2  of  2): 


life  cycle  estimates 

X 

estimates  for  program  initiation  only 

system  upgrades  (SCR)  software  change  requests  estimated  individually 

X 

systems  upgrades  estimated  as  w  hole 

estimates  for  on-gong  resources  needed  to  maintain  s/w 

X 

estimates  for  maintenance  not  done 

informal  re-estimates  during  development 

X 

formal  re-estimates  at  pre-defined  milestones 

formal  re-estimates  w  hen  amendment  changing  the  system  is  introduced 

X 

informal  re-estimates  w  hen  amendment  changing  the  system 

person  in-charge  of  estimation  w  alks  in  a  managers  office  to  get  an  opinion 

meeting(s)  organized  for  purpose  of  performing  cost  estimations 

factor  analysis  prior  to  commencement  of  program 

X 

none  done 

change  control  procedures  set  in  place 

X 

no  set  procedures 

elapsed  time  and  actual  work  time  estimates 

X 

one  or  the  other  or  neither 

no  schedule  created 

scheudle  created 

schedule  not  updated 

schedule  updated 

schedule  followed 

X 

schedule  not  followed 

tasks  identification  arises  as  program  progresses 

detailed  level  tasks  identified  prior  to  program  initiation 

scope  of  program  understood  by  all 

X 

scope  not  explicitly  defined 

quality  factors  and  criteria  identified 

X 

no  explicit  quality  factors  defined 

no  project  tracking  tools  used 

project  tracking  tools  used 

CSCIs  identified  and  tasked 

X 

CSCIs  not  explicitly  identified 

expectations  are  managed  via  estimations 

X 

estimations  are  made  to  fit  preconceived  expectations 

no  cost  schedule  developed 

cost  schedule  developed 

no  resource  schedule  developed 

resource  schedule  developed 

team  members,  management  know  at  any  time  if  in  budget  &  schedule 

X 

exact  budget  &  schedule  status  somew  hat  unclear  to  at  least  some 

individual  program  phases  are  estimated 

X 

only  top  level  program  estimated 

stakeholders/users  emphasis  understood-quick  to  field  or  all  complete 

X 

program  management  sets  delivery  tradeoffs  w  ithout  outside  input 

testing  planned  w  ith  initial  program  planning 

X 

testing  not  in  initial  planning 

documentation  not  considered  ininitial  planning 

X 

documentation  part  of  initial  planning 

hardware  considered  in  estimations 

X 

software  only  considered 

no  formal  schedule/cost  tracking 

formal  procedures  established  for  tracking  cost  and  schedule 

earned  value  set  up 

X 

earned  value  not  used 

estimations  omit  documentation  planning 

documentation  in  estimates 

training  omitted  in  estimates 

training  part  of  estimates 

earned  value  set  up,  but  not  tracked 

earned  value  tracked 

detailed  planning  done  w  ith  incomplete  set  of  requirements 

X 

detailed  planning  done  w  ith  detailed  set  of  requirements 

complete  infrastructure  support  mechanism  understood  for  estimations 

X 

no  consideration  of  infrastructure  done  for  estimations 

team  possibilities  considered  for  planning  of  program 

X 

no  consideration  for  outside  teaming  possibilities 

w  ork  breakdow  n  structure  (WBS)  set  up 

X 

no  WBS  completed 

Estimation/Planning  Management  (pg  2  of  2)  score  [32]  +pg  1  score  [32]  =  TOTAL  SCORE  [64]  Enter  on  QMM  scoresheet  blk  b. 
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Pair  choice  section  THREE:  (People  Management)  choose  most  applicable  term  of  the  two  for  each  row  (page  1  of  2): 
Human  Resources 


program  team  members  have  clearly  deined,  segmented  roles 


work  responsibilities  are  shared 


formal  team  building  procedures  are  used 


no  formal  team  building  emphasized 


program  manager  flexible  regarding  work  hours 


program  manager  maintains  strict  standards  for  work  hours 


big  picture  conveyed  to  all  team  members  by  program  management 


program  management  focuses  on  the  partitioned  tasks  with  team 


people  issues  dealt  with  primarily  through  indirect  methods  (email,  memo,  etc) 


people  issues  dealt  with  primarily  through  direct  methods  (face-to-face) 


training  is  required  and  planned  on  a  regular  basis 


graining  is  ad  hoc 


each  team  member  is  educated  on  and  understands  overall  program  and  their  roles 


| team  members  only  know  their  respective  areas 


consideration  for  team  members'  career  goals  are  reflected  in  assignments 


team  members  must  adapt  to  tasks  that  are  assigned 


team  members  assignments  and  responsibilities  are  mostly  dictated  by  PM 


assignments  and  responsibilities  are  discussed  and  agreed  upon  with  PM 


management  leads  in  problem  solving 


management  facilitates  and  lets  team  lead  in  problem  solving 


management  welcomes  problems  as  challenges  and  opportunities 


management  views  problems  as  obstacles  and  grounds  for  punishment 


team  members  participate  in  performance  evaluations  of  peers 


Personnel  evaluations  are  strictly  PM  responsibility 


management  reinforcement  feedback  sparse  and  inconsistent,  if  any 


management  provides  timely  reinforcement  feedback  for  positive  behaviors 


management  provides  basic  needs  of  office  facilities  fairly  well 


office  facilities  are  a  drawback  to  working  in  the  program 


working  conditions  are  fairly  comfortable,  time  off  policy  fairly  good 


working  conditions  and  time  off  policy  is  inconsistent  and  difficult  at  times 


Communication: 


communications  primarily  written  (email) 

X 

communications  primarily  verbal  (face-to-face) 

detailed  instructions:  oral  presentation,  follow-up  email 

X 

email  only 

formal  communication  protocol 

X 

informal  communications 

external  vertical  communications  restricted 

X 

external  vertical  communication  allowed 

coders  notebook  weekly  accomplishment  reports  required 

X 

not  required 

user-coder  relationship  established,  encouraged,  and  mediated 

X 

user-coder  interaction  minimized 

meetings  structured  to  minimize  waster  time 

X 

meetings  unstructured  and  open  ended 

meetings  have  agenda,  objectives,  and  conclude  with  action  items 

X 

meeting  agenda  fluid  and  open  ended 

program  management  and  coder  communication  face  to  face 

X 

program  management  and  coder  communication  primarily  email 

program  team  updated  regularly  regarding  organizational  &  program  status 

X 

meetings  infrequently  scheduled 

open  communications  is  encouraged 

X 

communication  hrough  chain  of  command  only  is  encouraged 

program  manager  accessible  for  discussions 

X 

program  manager  difficult  to  get  an  appointment  to  see 

program  management  (PM)  is  viewed  as  separate  from  team 

PM  mixes  with  team  frequently 

management  regularly  holds  team  meetings 

X 

meetings  are  sporadic 

meetings  are  structured  with  definite  goals  and  objectives 

X 

meetings  are  informal 

program  management  is  generally  easy  to  reach  and  talk  to 

X 

PM  is  usually  hard  to  get  a  hold  of  and  difficult  to  talk  to 

team-program  manager  relationship  adult-adult 

X 

team-program  management  relationship  parent-child 

schedules  are  spontaneous  and  poorly  communicated 

schedules  must  be  fixed  and  rigidly  followed  and  formally  reported 

work  is  seen  as  complex  processes  involving  team  working  together 

X 

work  broken  into  pieces  with  minimal  team  member  interaction 

action  items  often  is  poorly  disseminated  and  usually  not  followed  through 

action  items  communicated  and  followed  through  thoroughly 

team  members  require  frequent  clarifications  by  PM  for  assigned  tasks 

team  members  rarly  require  clarifications  by  PM  for  assigned  tasks 
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Pair  choice  section  THREE:  (People  Management)  choose  most  applicable  term  of  the  two  for  each  row  (page  2  of  2): 
Leadership: 


long  range  organizational  vision 

X 

short  tern  program  and  immediate  w  ork  focus 

lead  through  personal  attention  to  others 

X 

action-oriented  leadership  approach 

run  as  much  of  the  organization  as  possible 

let  team  make  decisions  as  much  as  possible 

direct  and  domineering  style 

encourage  independence  in  others 

traditional  leaders  respect  hierarchy 

do  w  hat  needs  to  be  done 

w  in  cooperation  rather  than  demand  it 

X 

tough-minded  with  others 

act  strongly  and  forcefully  in  the  field  of  ideas 

prefer  to  lead  other  independent  types  while  seeking  auto  no  my  for  self 

consults  w  ith  team  members  to  find  solutions  to  problems 

X 

consults  team  members  to  get  validation  of  PMs  predetermined  solutions 

keep  people  well  informed 

X 

only  as  much  know  ledge  as  necessary  for  their  w  ork 

make  things  happen  by  focusing  on  the  immediate  problems 

X 

long  range  focus  and  de-emphasize  current  problems 

manage  others  loosely  and  prefer  minimal  supervision 

X 

follow  traditional  procedures  and  rules  conscientiously 

leadership,  management  decisions  exclusively  by  program  management 

program  management  makes  decisions  but  gets  inputs  from  team 

team-program  manager  relationship  adult-adult 

X 

team-program  management  relationship  parent-child 

program  management  makes  decisions  but  gets  inputs  from  team 

all  program  team  members  responsible  for  program  decisions 

w  hen  a  problem  arises:  management  takes  over  to  solve  it 

management  lets  the  team  solve  the  problems 

leadership  is  do  as  1  say,  not  do  as  1  do 

X 

leadership  by  example 

program  expectation  not  influenced  by  PM 

X 

program  expectation  managed  by  PM 

PM  gives  freedom  to  team,  but  has  no  mentoring  for  members  (abdication) 

PM  empow  ers  teams  by  mentoring  members  to  be  leaders 

promgram  management  w  aits  and  sees  w  hat  happens  then  plans 

X 

management  plans  far  in  advance 

program  management  is  constantly  reacting  to  emergencies 

management  is  one  step  ahead  of  problems 

facilitative  approach  to  solving  problems 

X 

take  charge  readily  and  often 

program  management  is  complex,  takes  much  time  to  understand 

X 

management  is  simple,  easy  to  figure  out 

program  management  prefers  to  plunge  right  in 

takes  time  to  separate  things  to  be  done  and  order  of  doing  them 

program  management  reacts  spur  of  the  moment 

methodically  follows  plans 

Technical  Competency  of  the  Program  Manager: 


PM  has  technical  experience  particular  to  the  particular  s/w  program 

X 

PM  relies  on  team  members  solely 

PM  participates  in  technical  reviews 

X 

PM  only  in  non-technical  reviews 

PM  participates  in  making  technical  decisions  w  hen  problems  arise 

X 

PM  delegates  technical  questions 

PM  does  not  get  involved  discussing  technical  options 

PM  contributes  to  technical  options  being  discussed 

PM  does  not  review  technical  options  and  decisions 

PM  reviews  technical  options  and  decisions 

PM  actively  attempts  to  keep  up-to-date  with  current  technology  and  standards 

PM  is  removed  from  cutting  edge  technology  issues 

PM  receives  technical  periodicals  and  occasionally  references  applicable  articles 

X 

PM  doesn't  read  periodicals  nor  reference  current  articles  to  team 

PM  doesn't  have  technical  background  (or  education) 

X 

PM  has  technical  background  (or  education) 

team  members  avoid  PM  w  hen  they  need  technical  advice 

team  members  generally  consider  talking  to  PM  regarding  technical  issues 

HR  [1 3]  +  Comm.  [21]  +  Leadership  [20]  +  Tech.  Competency  [8]  =  People  Mgmt.  score  [60]  Enter  on  QMM  scoresheet  blk  c. 
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Pair  choice  section  FOUR:  (Risk  Management(RM))  choose  most  applicable  term  of  the  two  for  each  row  (page  1  of  2): 


RM  is  formal  and  documented 

X 

RM  is  informal,  if  at  all 

a  risk  management  plan  exists 

X 

no  risk  management  plan  is  developed 

RM  is  more  of  a  data  call  than  a  useful  document 

RM  drives  decisions  on  the  program 

X 

RM  is  done  prior  to  the  program  beginning 

RM  is  done  prior  and  during  program  execution 

X 

RM  is  only  done  during  the  program  execution 

RM  is  done  prior  and  during  program  execution 

X 

risks  are  generalized  through  the  whole  program 

risks  are  categorized 

X 

risk  management  is  done  internally,  only 

X 

an  outside  organization  also  contributes  to  the  RM  process 

risk  is  a  management  function 

X 

risk  is  a  program  team  function 

risks  are  precisely  articulated 

X 

risks  are  generalized,  if  at  all 

each  risk  has  a  consequence 

X 

consequences  are  generalized,  if  at  all 

a  mitigation  strategy  is  completed  for  each  risk 

X 

mitigation  strategy  is  generalized,  if  at  all 

contingency  plans  are  developed  for  a  RM  plan 

X 

contingency  plans  are  ad  hoc  as  problems  arise  in  the  program 

risks  are  anticipated 

X 

if  problems  arise,  management  will  deal  with  it 

the  program  doesn't  have  any  risk 

programs  that  do  not  have  risk,  have  problems 

X 

risk  management  is  automated 

X 

risk  management  may  use  tools,  but  depend  on  human  input 

risks  are  assigned  probabilities 

X 

probabilities  are  not  relevant  for  RM 

all  risks  are  potential  problems,  relative  priorities  for  risks  are  not  useful 

X 

risks  are  weighed  relative  to  other  program  risks  and  thus  prioritized 

risk  management  information  is  only  shared  internally 

X 

risk  management  information  is  shared  with  all  stakeholders 

risk  analysis  uses  ordinal  rankings 

X 

risk  analysis  uses  actual  measurements  with  a  mathematical  model 

regret  analysis  used 

X 

no  regret  analysis  done 

attach  probabilities  to  future  events 

X 

no  probabilities  associated  with  future  events 

assessing  risks  with  mechanical  meethods 

X 

risks  should  be  compared  to  other  risks  and  sorted 

X 

risk  status  tracked 

X 

not  tracked 

technical  risks  examined 

X 

no  technical  risks  examined 

process  risks  examined 

X 

no  process  risks  examined 

product  risks  examined 

X 

no  product  risks  examined 

stakeholder/user  risks  examined 

X 

no  examination  of  stakeholder/user  risks 

checklists  used  to  identify  risks 

X 

no  checklists  used 

risks  are  tracked 

X 

no  tracking  or  monitoring  of  risks 

each  risk  has  an  impact 

X 

no  impact  analysis  of  risk 

each  risk  has  a  mitigation  plan 

X 

no  individual  risk  mitigation 

risks  monitored  by  priority 

X 

no  special  attention  to  track  higher  priority  risks 

risk  assessment  is  formalized 

X 

no  formal  risk  assessment 

risk  control  is  formalized 

X 

no  formal  risk  control 

integration  risks  not  considered 

integration  risks  examined 

X 

Risk  Management  (page  1  of  2)  score 

30 
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Pair  choice  section  FOUR:  (Risk  Management(RM))  choose  most  applicable  term  of  the  two  for  each  row  (page  2  of  2): 


risks  to  cost 

X 

no  cost  risks  examined 

unforeseen  risks  have  occurred  in  program 

any  risk  that  came  up  had  been  identified  previously 

X 

personnel  risks  examined 

X 

no  personnel  risks  examined 

estimation  risks  examined 

X 

no  estimation  risks  examined 

planning  risks  examined 

X 

no  planning  risks  examined 

requirements  risks  examined 

X 

no  requirements  risks  examined 

resource  risks  examined 

X 

no  resource  risks  examined 

risk  management  plan  updated  regularly 

X 

no  regular  risk  management  plan  updates 

risks  charted 

X 

risks  not  charted 

performance  risks  examined 

X 

performance  risks  not  examined 

program  management  self  risks  examined 

X 

no  program  management  risks  examined 

risk  from  program  constraints  examined 

no  program  constraint  risks  examined 

X 

each  category  of  risks  are  prioritized 

X 

no  prioritization 

each  category  of  risks  are  evaluated  for  impact 

X 

no  impact  analysis  performed 

each  category  of  risks  have  control  strategy 

X 

no  control  strategy 

documentation  risks  examined 

X 

no  documentation  risks  examined 

regret  matrix  tracked 

X 

no  regret  matrix  or  not  tracked 

communication  of  risk  activities  are  facilitated 

X 

no  facilitation  or  promotion  of  communication  of  risk  activities 

taxonomy-based  questionnaire  used  to  identify  risks 

taxonomy-based  questionnaire  not  used 

X 

associated  hardware  risks  examined 

X 

no  consideration  for  hardware  risks 

integration  risks  examined 

X 

integration  risks  not  examined 

communication  risks  examined 

X 

communication  risks  not  examined 

leadership  risks  examined 

leadership  risks  not  considered 

X 

risk  avoidance  considered  for  certain  risks 

X 

risk  avoidance  not  considered  for  risks 

risk  documentation  forms  used 

X 

no  risk  documentation  forms  used 

dependency  risks  examined 

X 

no  dependency  risks  examined 

alternatives  like  risk  avoidance  considered  for  high  risk  items 

X 

no  consideration  of  risk  avoidance 

documented  risk  statements  use  a  condition-consequence  type  format 

X 

condition-consequence  of  risk  statements  not  clearly  defined 

no  assignment  of  ownership  of  risk  mitigation  action 

each  risk  mitigation  action  is  assigned  to  an  individual  for  resolution 

X 

calculation  of  risk  exposure  made  (probability  X  loss,  for  each  risk) 

no  risk  exposure  calculations 

X 

oral  communication  of  risks  only 

risks  written  in  a  way  that  communicates  nature  and  status  of  factors 

X 

triggers  used  to  quantify  risk  conditions  present 

X 

risk  conditions  present  are  all  subjective 

risk  "czar'’  in  program  for  monitoring  risks 

no  special  positions/responsibilities  for  risk  monitoring 

X 

post-program  review  completed  (scheduled)  for  unanticipated  problems  ID 

no  post-program  reviews  completed  or  scheduled 

X 

no  schedule  risks  examined 

risks  to  schedule  investigated 

X 

Risk  Management  (pg  2  of  2)  score  [30]  +pg  1  score  [31]  =  TOTAL  SCORE  [61]  Enter  on  QMM  scoresheet  blk  d. 
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E.  PROGRAM  C  -  PROGRAM  MANAGER 
1.  QMM  Summary  Score  Sheet 


QMM  Scoresheet 

Part  One 

Part  Two 

Total 

Importance 

Weighted 

Category 

Score 

Score 

Score 

Coefficient 

Score 

Requirements 

Management 

a 

46 

e 

7 

53 

0.92 

= 

48.76 

Est./Planning 

Management 

D 

60 

D 

44 

104 

0.67 

= 

69.68 

People 

Management 

c 

18 

g 

-15 

3 

1.86 

= 

5.58 

Risk  Management 

D 

62 

D 

47 

109 

0.55 

= 

59.95 

QMM  SCORE  183.97 


Max.  QMM  score  possible  528.00 

Min.  QMM  score  possible  -130.86 

QMM  percentage  score:  47.78% 


Objective/Subjective  view  of  the  overall  success  of  program  A  on  a  scale  of  0  to  10 
(0  being  total  failure,  10  being  perfect  program  total  success) 

Survey  Participant:  Program  Manager 

Success  Score:  6 
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2.  Requirements  Management  Questionnaire  Responses 


No.  Requirements  Management  Questionnaire  -  Total:  Block  e  Yes  No  N/A 


1 

PM  chose  to  have  a  formal  requirements  list 

X 

2 

Requirements  recorded  in  some  way 

X 

3 

Written  requirements  were  part  of  some  formal  document 

X 

4 

Written  requirements  were  informal 

X 

5 

At  least  some  requirements  were  oral  only 

X 

6 

All  stakeholders  were  identified 

X 

7 

All  stakeholders  participated  in  the  requirements  extraction 

X 

8 

Some  stakeholders  participated  in  the  requirements  extraction 

X 

9 

Management  extracted  requirements,  no  stakeholder  involvement 

X 

10 

Management  passed  requirements  to  development  team 

X 

11 

Stakeholders  not  involvved  in  Management  extraction,  but  approved 

X 

12 

Management  gets  inputs  from  stakeholders,  then  develops  requirements 

IT 

13 

Developers  work  informally  with  users  to  arrive  at  requirements 

X 

14 

Same  as  13,  but  management  oversees  and  formalizes 

X 

If  a  waterfall  or  sequential  development  strategy:  | 

15 

All  requirements  complete  before  design 

16 

Some  requirements  left  incomplete  prior  to  design 

17 

Requirements  informal  prior  to  design  effort 

18 

Requirements  serve  as  input 

19 

Length  of  time  for  requirements  w  ork  greater  than  development  w  ork 

20 

Requirements  developed  in  parallel  to  design 

OR  If  a  prototype,  throwaway,  or  other  development  strategy:  [ 

15 

Learn  about  requirements  through  development  efforts 

X 

16 

No  coding  until  all  requirements  are  defined 

X 

17 

Requirements  formal  prior  to  design  effort 

X 

18 

Requirements  serve  as  output 

X 

19 

Requirements  definition  work  in  parallel  to  development  efforts 

X 

20 

Requirements  developed  in  parallel  to  design 

“X- 

21 

Are  requirements  frozen  at  some  phase 

X 

22 

Change  management  exists 

X 

23 

Change  management  is  formal 

X 

24 

Project  strategy  is  consistent  throughout  development 

X 

25 

Requirements  are  updated 

X 

26 

Configuration  Management  (CM)  exists 

“x- 

27 

CM  is  formal 

X 

28 

Requirements  are  testable 

X 

29 

Requirements  testing  considered/implemented  during  extraction 

X 

30 

Requirements  testing  plan  exists 

X 

31 

Requirements  testing  is  formal 

X 

32 

All  requirements  have  priorities 

X 

33 

All  requirements  must  be  implemented 

X 

34 

Requirements  are  tested 

X 

35 

All  requirements  are  equally  important 

X 

36 

At  least  some  requirements  have  priorities 

X 

37 

All  requirements  are  traceable 

X 

38 

Traceability  not  important 

X 

39 

Each  requirement  has  an  author 

X 

40 

Who  authored  requirement  is  not  important 

X 

41 

Initial  set  of  requirements  to  be  implemented,  no  requirements  creep 

X 

42 

Structured  and  tracked  changes  to  requirements  only 

X 

43 

Change  is  inevitable,  changes  allow  ed  at  all  times 

X 

44 

Change  is  inevitable,  but  changes  limited 

X 

45 

Requirements  control  funding 

X 

46 

Requirements  history  kept 

X 

47 

Baseline  established  for  requirements  at  some  point  prior  to  develop 

TOTAL  SCORING 

~7~ 
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3.  Estimation/Planning  Questionnaire  Responses 


No.  Estimation/Planning  Questionnaire  -  Total:  Block  f  Yes  No  N/A 


1 

Avolume  product  metric  used  (LOC,  #  of  files,  #  of  screens,  pages  of  doc) 

X 

Total 

2 

Measure  used  for  various  product  elements  (modules,  components,  CSCI) 

X 

3 

Product  measures  made  by  phase  (amt  at  implementation,  LOC  changed  at  unit  test) 

X 

4 

Other  product  attributes  measured  (FP,  throughput,  mem  cap,  cyclomatic  complexity) 

X 

5 

Product  matrics  tracked  and  updated  hroughout  program  execution 

X 

6 

Event  count  process  metric  used  (#  defects  in  test,  reqmt  changes,  milestones  met) 

X 

7 

Time  measure  process  metric  used  (cycle  time) 

X 

8 

Process  metrics  tracked  and  updated  throughout  program  execution 

X 

9 

Program  cost  estimations  made  from  product  or  process  metrics 

X 

10 

Program  costextimations  tracked  and  updated  to  reflect  progress/changes 

X 

11 

Factor  analysis  performed  on  program 

X 

12 

Program's  primary  purpose,  including  major  functions  and  deliverables  known 

X 

13 

Work  breakdown  structure  developed 

X 

14 

Task  estimated  with  realistic  expectations  of  productivity  probabilities 

X 

15 

Schedules  developed  based  on  realistic  expectations 

X 

16 

Schedules  tracked  and  updated  based  on  new  information 

X 

17 

Detailed  activity  lists  used  for  clearly  defined  completed/not  completed  tasks 

X 

18 

Quality  assurance  plan  orsimilarto  aid  in  detecting  defects  earlyin  program 

X 

19 

COCOMO  estimates  performed 

X 

20 

CSCI  clearly  defined  and  tasked 

X 

21 

Estimates  completed  ad  hoc 

X 

22 

Gantt  charts  used  and  updated 

X 

23 

Resource  estimations  (working  hrs,  job  categories,  task  activities)  done 

X 

24 

Earned  value  established 

X 

25 

Earned  value  tracked  throughout  program 

X 

26 

Quality  expectations  established  for  product  with  users  and  stakeholders 

X 

27 

Critical  path  for  program  tasks  developed  and  tracked 

X 

28 

Measure  of  effectiveness  (MOE)  or  Figure  of  merit  established  and  tracked 

X 

29 

Estimates  are  updated  routinely 

X 

30 

Schedules  are  updated  routinely 

X 

31 

Estimations  are  made  by  program  management  (top-down) 

X 

32 

Estimateions  are  made  by  program  team  members  (bottom-up) 

X 

33 

Automated  program  tracking  used 

X 

34 

PM  usually  thorough  in  tracking  and  reporting  schedules  and  financials 

X 

35 

WBS  developed  only  as  data  call 

X 

36 

Earned  value  used  to  track  program  progress 

X 

37 

PM  insists  on  prioritizing  work  reduction  as  schedule/funding  compromised  by  stakeholders 

X 

38 

Estimations  are  done  using  both  top  down  and  bottoms  up  approaches 

X 

39 

All  program  team  members  involved  in  planning  process 

X 

40 

Hardware  also  considered  in  estimaation  process 

X 

41 

Program  history  compiled 

X 

42 

System  upgrades  (SCR)  software  changes  requests  estimated  individually 

X 

43 

Management  duties  apart  of  each  team  member's  responsibilities 

X 

44 

PM  dictates  schedules  to  program  team 

X 

45 

Code  reviews  planned  in  schedule 

X 

46 

Defined  tangible  milestones  established  for  program  tasks 

X 

47 

Test  planning  done  at  the  start  of  the  program 

X 

48 

Estimations  are  completed  bythose  performing  the  tasks 

X 

49 

Sensitivity  analysis  performed  for  program  choices 

X 

X 

50 

Software  deployment  planning  completed 

X 

X 

TOTAL  SCORING 

□ 

44 1 
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4.  People  Management  Questionnaire  Responses 


No.  People  Management  Questionnaire  -  Total:  Block  g 

Yes 

No 

N/A 

1 

PM  is  accessible  in  person  by  each  team  member 

X 

2 

PM  is  accessible  via  email  (memo,  letter)  by  each  team  member 

X 

3 

PM  is  accessible  via  phone  by  each  team  member 

X 

4 

PM  not  only  considers  a  person's  suitability,  not  also  desire  to  be  on  a  team 

X 

5 

PM  consults  with  each  team  member  regarding  their  career  goals 

X 

6 

PM  regularly  holds  meetings  to  inform  team  of  program  progress 

X 

7 

PM  solicits  opinions  from  team  members  before  making  decisions 

X 

8 

PM  lets  teams  make  decisions  affecting  their  work 

X 

9 

PM  freuently  makes  decisions  without  any  consultation  with  members 

X 

10 

PM  understands  the  technology/language  of  the  program 

X 

11 

PM  is  able  to  communicate  with  other  the  technical  issues  in  the  program 

X 

12 

PM  prioritized  problems  or  conflicts  within  the  program 

X 

13 

PM  assists  team  members  in  developing/advising  of  career  path 

X 

14 

PM  empowers  program  members  to  recommend  hiring  new  team  members 

X 

15 

PM  empowers  program  members  to  recommend  firings  of  other  members 

X 

16 

PM  specifically  assigns  work  to  each  program  member 

X 

17 

PM  sets  communication  protocol 

X 

18 

PM  allows  unrestricted  communications 

X 

19 

PM  encourages  or  requires  training  for  each  individual 

X 

20 

PM  takes  control  in  difficult/roblem  areas 

X 

21 

PM  looks  ahead  to  new  programs,  new  upgrades  of  existing  program 

X 

22 

PM  maintains  regular  communications  with  all  stakeholders 

X 

23 

PM  maintains  regular  communications  with  users 

X 

24 

PM  encourages  program  team  communication  with  users 

X 

25 

PM  encourages  program  team  communication  with  stakeholders 

X 

26 

PM  facilitates  horizontal  communication  within  program 

X 

27 

PM  facilitates  communication  during  integration 

X 

28 

PM  holds  meetings  without  clear  objectives 

X 

29 

PM  must  approve  all  decisions  within  the  program 

X 

30 

PM  must  approve  all  interactions  with  stakeholders 

X 

31 

PM  must  approve  all  interactions  with  users 

X 

32 

PM  makes  all  presentations  to  stakeholders/users 

X 

33 

PM  is  considered  "flexible"  in  terms  of  program  members  personal  issues 

X 

34 

PM,  at  least  occasionally,  schedules/promotes  outside  work  team  activities 

X 

35 

PM  is  readily  willing  to  listen  to  program  prblems  and  complaints 

X 

36 

PM  takes  action  to  resolve  program  problems  and  complaints 

X 

37 

PM  is  generally  respected  by  stakeholders,  users,  and  organization 

X 

38 

PMsometimes  fails  to  grasp  important  technical  issues  in  program 

X 

39 

PM  recruits  program  team  members  from  outside  organization 

X 

40 

PM  participates  in  technical  reviews 

X 

41 

Program  personnel  have  clearly  defined  specific  tasks 

X 

42 

Although  individual's  tasks  are  specific,  each  exposed  to  the  "bigger  picture" 

X 

43 

PM  has  clearly  defined  his/her  expectations  for  each  individual 

X 

44 

PM  delegation  of  duties  is  usually  seemless  in  execution 

X 

45 

PM  acts  as  facilitator  to  solving  personnel  conflicts 

X 

46 

PM  attempts  to  motivate  individuals  on  the  program  team 

X 

47 

PM  clearly  spearates  technical  from  managerial  roles  for  individuals 

X 

48 

PM  directs  how  he/she  expects  the  task  to  be  accomplished 

X 

49 

PM  directs  what  needs  to  be  done,  but  does  not  direct  how 

X 

50 

PM  attempts  to  spotlight  individuals  in  the  program  for  positive  exposure 

X 

TOTAL  SCORING 

-4 

-11 
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5. 


Risk  Management  Questionnaire  Responses 


No.  Risk  Management  Questionnaire -Total:  Block  h  Yes  No  N/A 


1 

Risk  Management  (RM)  is  specifically  an  activity  in  the  program 

X 

Total 

2 

RMis  formal  and  documented 

X 

3 

Aspecific  RM  Ian  exists 

X 

4 

RMis  required  in  the  program,  but  not  used  during  the  program 

X 

5 

RMis  done  prior  to  the  program  execution 

X 

6 

RM  is  done  by  an  outside  entity  to  the  development 

X 

7 

RMis  done  internally  only 

X 

8 

RMis  both  internally  performed  and  externally  assessed 

X 

9 

RM  planning  occurs  during  or  after  major  milestones  in  the  program 

X 

10 

RiskAssessmentis  onlya  management  function 

X 

11 

RMis  informal  or  non  existent 

X 

12 

There  is  a  RM  plan,  but  it  is  not  updated  or  tracked 

X 

13 

Risks  are  only  generalized 

X 

14 

Each  risk  is  delineated 

X 

15 

Each  risk  has  a  consequence 

X 

16 

Each  risk  has  a  likelihood  rating  of  some  sort 

X 

17 

Each  risk  has  a  mitigation  strategy 

X 

18 

Risk  Management  is  automated 

X 

19 

Risks  are  tracked 

X 

20 

21 

Regret  analysis  performed 

X 

22 

RM  drives  decisions  in  the  program 

X 

23 

Risks  have  probabilities 

X 

24 

Risk  Management  is  ad  hoc 

X 

25 

RM  information  is  shared  with  all  stakeholders  (as  appropriate) 

X 

26 

Risks  are  weighed  relative  to  other  program  risks 

X 

27 

RiskAssessmentis  a  program  team  activity 

X 

28 

Risk  Assessment  done  prior  to  program  start 

X 

29 

Risk  Assessment  includes  personnal  risk 

X 

30 

RM  uses  tools,  but  depends  on  human  decisions 

X 

31 

Risk  assessment  includes  cost  risks 

X 

32 

Risk  Assessment  includes  schedule  risks 

X 

33 

Risk  Assessment  includes  technology  risks 

X 

34 

RiskAssessmentis  briefed  organization  structure  above  program  manager 

X 

35 

Risk  Assessment  includes  requirements  risks 

X 

36 

Risk  Assessment  includes  user  risks  (too  little  involvement  of  user) 

X 

37 

Risk  Assessment  includes  documentation  risks 

X 

38 

Risk  Assessment  includes  integration  risks 

X 

39 

Risk  Assessment  includes  interface  risks  (non-standard) 

X 

40 

Risk  Assessment  includes  continuing  requirements  change  (feature  creep) 

X 

41 

Risk  Assessment  includes  dependent  projects/programs  risks 

X 

42 

Documentation  proof  exists  to  demonstrate  following  risk  management  plan 

X 

43 

High  rish  have  measured  tracking  (high  profile  status) 

X 

44 

Organizational  history  used  to  search  for  risks 

X 

45 

Other  organizational  checklists  used  for  risk  assessment 

X 

46 

Internal  organizational  checklists  used  for  risk  assessment 

X 

47 

Risk  Assessment  information  contributed  to  internal  or  other  database 

X 

48 

Risk  Assessment  includes  internal  organization  risks 

X 

49 

Risk  Assessment  includes  stakeholder  risks 

X 

50 

No  risk  management  needed;  program  is  straightforward  &  understood 

X 

TOTAL  SCORING 

m 

_47j 
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6.  Pair  Choices  Responses 

Pair  choice  section  ONE:  (Requirements  Management)  choose  most  applicable  term  of  the  two  for  each  row  (page  1  of  2): 


formal  requirement  list 

X 

informal  requirement  list 

w  ritten  requirements 

X 

oral  requirements 

requirements  informal,  but  recorded 

X 

requirements  not  recorded 

requirements  as  part  of  an  SRS  (or  other  formal  repository) 

X 

requirements  informally  recorded 

requirements  taken  as  is  from  customer 

look  to  reformulate,  interview  in-depth,  or  otherwise  re-validate 

X 

only  one  development  strategy  used 

X 

strategies  not  consistent,  used  at  different  times 

stakeholders  as  part  of  requirements  development 

X 

stakeholders  approving  requirements  after  formulated  by  development  team 

requirements  are  testable 

X 

requirements  have  no  test  plans 

informal  test  plan  or  no  test  plan 

formal  test  plan 

X 

test  team  involved  w  ith  requirements 

X 

no  test  team  input  or  plans  during  requirements  development 

only  a  percentage  of  requirements  present  in  baseline 

baseline  must  contain  all  requirements 

X 

requirements  documentation  has  hierarchical  structure 

X 

all  requirements  must  be  implemented 

requirements  have  listed  responsible  party 

X 

requirements  origin  not  important 

requirements  documentation  have  versions 

X 

no  requirements  history 

requirements  have  specific  attribute  values 

X 

requirements  all  rank  evenly 

funding  controls  requirements  definition 

requirements  definition  controls  funding 

X 

reqquirements  are  top  dow  n 

X 

requirements  are  bottom  up 

users/stakeholders  are  identified  and  interviewed  (market  survey) 

no  special  consideration  to  identify  users/stakeholders 

X 

each  requirement  has  a  singular  concept 

some  requirements  are  compound  statements 

X 

requirements  definition  minimized  when  funding  short 

X 

program  scope  may  reduce,  but  requirements  definition  completed 

requirements  extraction  has  formal  process 

X 

requirements  extraction  ad  hoc 

change  procedures  formal 

X 

change  procedures  ad  hoc 

users/stakeholders  somehow  involved  in  requirements  definition 

X 

program  team  only  involved  in  requirement  definition 

management  sets  requirements  for  developers 

developers  at  least  partially  involved  in  setting  requirements 

X 

requirements  changed  at  least  once  since  baseline  established  prior  to  newversion 

X 

requirements  in  baseline  has  not  changed  prior  to  newversion  or  upgrade 

no  ranking  of  requirements 

requirements  have  priorities  assigned 

X 

use-case  diagrams  (or  other  models  or  scenario  developments) 

X 

no  models  used  for  requirements  extraction 

requirements  changes  informal 

requirements  changes  formal 

X 

plan  to  "freeze"  requirements  at  some  designated  milestone 

no  provision  for  "freezing"  requirements 

X 

requirements  must  be  traceable 

X 

origin  of  requirements  not  important 

requirements  must  be  testable 

system  developed  must  be  testable 

X 

test  plans  to  determine  requirements  implemented 

no  test  plans  needed  for  requirements  verification 

X 

requirements  have  priorities  in  implementation 

X 

all  requirements  must  be  implemented 

some  requirements  have  multiple  statements  or  ideas 

X 

one  idea,  one  statement  per  requirement 

Requirements  Management  (page  1  of  2)  score  |  29 
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Pair  choice  section  ONE:  (Requirements  Management)  choose  most  applicable  term  of  the  two  for  each  row  (page  2  of  2): 


|  AN  S  WE  R  THIS  BLOCK  OF  QUESTIONS  ONLY  IF  A  SEQUENTIAL  OR  WAT  ERF  ALL  AP  P  ROACH  IS  USED  FOR  DEVELOP  M  ENT  (Requirements  page  2  of  2)  [ 

requirements  first,  then  initial  development  w  ork 

initial  development  w  ork  then  requirements 

requirements  documentation  driving  development 

requirements  documentation  developed  in  parallel/after  development 

user  feedback  considered  during  development 

after  development  starts,  user  feedback  serves  as  input  to  new  work 

change  management  procedures  used  strictly 

change  management  procedures  as  guidance  only 

design  decisions  prior  to  or  in  parallel  to  requirrements  development 

design  decisions  only  after  approved  requirements  stabilized 

requirements  summarized  w  ht  w  e  have  developed 

requirements  are  the  blueprint  for  development 

length  of  time  for  requirements  w  ork  greater  than  development  w  ork 

length  of  time  for  requirements  work  less  than  development  w ork 

requirements  have  design  detail 

no  design  detail  in  requirements 

requirements  creep  to  be  avoided 

requirements  creep  o.k.,  but  need  to  be  controlled 

freeze  requirements  at  some  point 

requirements  are  fluid  throughout  development 

formal  change  procedure 

informal  change  procedure 

change  management  plan 

no  change  management  plan 

requirements  ambiguity  always  present  to  some  extent 

requirements  ambuiguity  unacceptable  at  any  level 

testing  considered  up  frornt  during  requirements  determination 

testing  considered  down  the  line  during  development 

requirements  development  team  members  different  from  implementation 

those  working  on  requirements,  work  on  implementation 

start  implementation  as  early  as  possible  to  help  define  requirements 

requirements  must  be  defined  prior  to  any  implementation  work 

ANSWER  THIS  BLOCK  OF  QUESTIONS  ONLY  IF  A  PROTOTYPING,  THROWAWAY,  SYNCHRONIZE  &  STABILIZE,  OR  OTHER  STRATEGY  USED 

develop  prototype,  then  determine  requirements 

X 

determine  requirements  prior  to  any  development  work 

requirements  testing  done  after  each  iteration 

X 

no  testing 

individual  changes  as  necessary 

X 

only  block  changes  made 

development  team  decides  on  changes  after  iteration 

users  involved  w  ith  changes 

X 

changes  based  on  feedback  only  from  user  for  correction  of  problems 

changes  to  upgrade  system  and  correct  problems 

X 

funding  controls  changes  and  change  procedures 

X 

changes  control  funding 

requirements  documentation  finalized  prior  to  development 

requirements  fluid  throughout  development  (only freeze  at  end) 

X 

requirements  test  plans  completed  prior  to  development 

X 

requirements  test  plans  completed  after  development 

requirements  first,  then  initial  development  w  ork 

X 

initial  development  work  then  requirements 

use  development  effort  to  learn  more  about  requirements 

X 

define  all  requirements  prior  to  coding  anything 

requirements  ambiguity  always  present  to  some  extent 

X 

requirements  ambiguity  unacceptable  at  any  level 

requirements  have  design  detail 

X 

no  design  detail  in  requirements 

user  feedback  considered  during  development 

X 

after  development  starts,  user  feedback  serves  as  input  to  new  work 

get  something  to  users  as  soon  as  possible  for  evaluation 

X 

make  sure  it  is  complete  before  releasing 

management  dictates  requirements 

X 

development  team  visually  represent  requirements  through  rapid  prototyping 

new  requirements  allowed  after  initial  requirements  defined 

X 

new  requirements  not  allow  ed 

Requirements  Management  (pg  2  of  2)  score  [1 7]  +pg  1  score  [29]  =  TOTAL  SCORE  [46]  Enter  on  QMM  scoresheet  blk  a. 
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Pair  choice  section  TWO:  (Estimation/Planning  Management)  choose  most  applicable  term  of  the  two  for  each  row  (page  1  of  2): 


at  least  one  estimation  method  used  in  program 

X 

no  estimates 

formal  derivation  of  product  metric  for  estimation  of  size 

X 

ad  hoc  size  estimation 

ad  hoc  process  evaluation 

formal  derivation  of  at  lest  one  process  metric 

develop  work  breakdown  structure  (WBS) 

X 

assign  work  as  needs  arise 

estimates  are  developed  to  fulfill  a  data  call  only 

use  estimates  to  plan  program 

use  estimates  to  sell  program  only 

estimates  are  useful  to  the  project  tema  for  planning  purposes 

resource  evaluations  made  for  program 

X 

no  resource  evaluation  for  planning 

use  both  bottom  up  &  top  down  for  estimate,  use  one  stakeholders  like 

use  both  bottom  up  &  top  down  and  evaluate  significant  differences 

estimates  made  and  not  updated 

estimates  updated  throughout  program 

resources  estimations  used  to  adjust  product  size  estimate 

X 

estimations  made  irregardless  of  resources  available 

estimations  made  to  fit  budget 

budget  made  from  estimations 

estimations  compromised  to  get  program 

rather  risk  loss  of  program  than  compromise  confident  estimations 

cycle  time  estimations 

X 

no  cycle  time  estimations 

event  count  estimations 

X 

no  event  count  estimations 

lines  of  code  (LOC)  estimation 

X 

no  LOC  estimation 

function  pont  (FP)  estimation 

X 

no  FP  estimation 

estimates  by  algorithmic  methods 

X 

estimates  by  analogy 

expert  judgement  for  estimates 

X 

ad  hoc  estimates 

estimates  by  algorithmic  methods 

X 

ad  hoc  estimates 

expert  judgement  for  estimates 

X 

estimates  by  analogy 

ad  hoc  estimates 

estimates  by  analogy 

bottom  up  estimates 

X 

expert  judgement 

top  down  estimates 

X 

expert  judgement 

ad  hoc  estimates 

any  other  estimate  process 

fuzzy  logic  estimating  method 

X 

no  formal  estimation  methodology 

WBS  development  from  estimates 

X 

WBS  development  in  parallel  or  prior  to  estimation  completion 

critical  path  of  program  determined 

X 

tasks  developed  but  no  path  is  identified 

estimators  are  program  team  members 

X 

estimators  are  outside  program  team 

management  only  on  estimations 

X 

all  team  members  involved  in  estimation  process 

estimates  updated  at  reviews 

X 

no  updates  of  estimates 

estimates  updated  at  reviews 

X 

estimates  constantly  updates  (in  between  reviews,  to) 

estimate  procedures  stay  the  same 

X 

estimate  procedures  change 

stakeholders  are  part  of  estimation  process 

X 

stakeholders  brief  estimations  after  completion 

estimates  are  used  beyond  initial  selling  of  program 

X 

estimates  are  one  time  events,  used  for  a  specific  purpose  once 

WBS  has  objective  measure  of  completeness 

X 

important  to  have  WBS  as  guide,  not  rigid  implementation 

Estimation/Planning  Management  (page  1  of  2)  score  [ 

1  33  1 
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Pair  choice  section  TWO:  (Estimation/Planning  Management)  choose  most  applicable  term  of  the  two  for  each  row  (page  2  of  2): 


life  cycle  estimates 

X 

estimates  for  program  initiation  only 

system  upgrades  (SCR)  software  change  requests  estimated  individually 

X 

systems  upgrades  estimated  as  w  hole 

estimates  for  on-gong  resources  needed  to  maintain  s/w 

X 

estimates  for  maintenance  not  done 

informal  re-estimates  during  development 

formal  re-estimates  at  pre-defined  milestones 

formal  re-estimates  w  hen  amendment  changing  the  system  is  introduced 

X 

informal  re-estimates  w  hen  amendment  changing  the  system 

person  in-charge  of  estimation  w  alks  in  a  managers  office  to  get  an  opinion 

X 

meeting(s)  organized  for  purpose  of  performing  cost  estimations 

factor  analysis  prior  to  commencement  of  program 

none  done 

change  control  procedures  set  in  place 

X 

no  set  procedures 

elapsed  time  and  actual  w  ork  time  estimates 

X 

one  or  the  other  or  neither 

no  schedule  created 

scheudle  created 

schedule  not  updated 

schedule  updated 

schedule  followed 

X 

schedule  not  followed 

tasks  identification  arises  as  program  progresses 

X 

detailed  level  tasks  identified  prior  to  program  initiation 

scope  of  program  understood  by  all 

X 

scope  not  explicitly  defined 

quality  factors  and  criteria  identified 

no  explicit  quality  factors  defined 

no  project  tracking  tools  used 

project  tracking  tools  used 

CSCIs  identified  and  tasked 

CSCIs  not  explicitly  identified 

expectations  are  managed  via  estimations 

X 

estimations  are  made  to  fit  preconceived  expectations 

no  cost  schedule  developed 

cost  schedule  developed 

no  resource  schedule  developed 

resource  schedule  developed 

team  members,  management  know  at  any  time  if  in  budget  &  schedule 

X 

exact  budget  &  schedule  status  somew  hat  unclear  to  at  least  some 

individual  program  phases  are  estimated 

X 

only  top  level  program  estimated 

stakeholders/users  emphasis  understood-quick  to  field  or  all  complete 

X 

program  management  sets  delivery  tradeoffs  w  ithout  outside  input 

testing  planned  w  ith  initial  program  planning 

X 

testing  not  in  initial  planning 

documentation  not  considered  ininitial  planning 

documentation  part  of  initial  planning 

hardware  considered  in  estimations 

software  only  considered 

no  formal  schedule/cost  tracking 

formal  procedures  established  for  tracking  cost  and  schedule 

earned  value  set  up 

X 

earned  value  not  used 

estimations  omit  documentation  planning 

documentation  in  estimates 

training  omitted  in  estimates 

training  part  of  estimates 

earned  value  set  up,  but  not  tracked 

earned  value  tracked 

detailed  planning  done  w  ith  incomplete  set  of  requirements 

detailed  planning  done  w  ith  detailed  set  of  requirements 

complete  infrastructure  support  mechanism  understood  for  estimations 

X 

no  consideration  of  infrastructure  done  for  estimations 

team  possibilities  considered  for  planning  of  program 

X 

no  consideration  for  outside  teaming  possibilities 

w  ork  breakdow  n  structure  (WBS)  set  up 

X 

no  WBS  completed 

Estimation/Planning  Management  (pg  2  of  2)  score  [27]  +pg  1  score  [33]  =  TOTAL  SCORE  [60]  Enter  on  QMM  scoresheet  blk  b. 
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Pair  choice  section  THREE:  (People  Management)  choose  most  applicable  term  of  the  two  for  each  row  (page  1  of  2): 
Human  Resources 


program  team  members  have  clearly  deined,  segmented  roles 

X 

work  responsibilities  are  shared 

formal  team  building  procedures  are  used 

no  formal  team  building  emphasized 

X 

program  manager  flexible  regarding  work  hours 

program  manager  maintains  strict  standards  for  work  hours 

X 

big  picture  conveyed  to  all  team  members  by  program  management 

program  management  focuses  on  the  partitioned  tasks  with  team 

X 

people  issues  dealt  with  primarily  through  indirect  methods  (email,  memo,  etc) 

people  issues  dealt  with  primarily  through  direct  methods  (face-to-face) 

X 

training  is  required  and  planned  on  a  regular  basis 

X 

training  is  ad  hoc 

each  team  member  is  educated  on  and  understands  overall  program  and  their  roles 

team  members  only  know  their  respective  areas 

X 

consideration  for  team  members'  career  goals  are  reflected  in  assignments 

team  members  must  adapt  to  tasks  that  are  assigned 

X 

team  members  assignments  and  responsibilities  are  mostly  dictated  by  PM 

assignments  and  responsibilities  are  discussed  and  agreed  upon  with  PM 

X 

management  leads  in  problem  solving 

X 

management  facilitates  and  lets  team  lead  in  problem  solving 

management  welcomes  problems  as  challenges  and  opportunities 

management  views  problems  as  obstacles  and  grounds  for  punishment 

X 

team  members  participate  in  performance  evaluations  of  peers 

Personnel  evaluations  are  strictly  PM  responsibility 

X 

management  reinforcement  feedback  sparse  and  inconsistent,  if  any 

X 

management  provides  timely  reinforcement  feedback  for  positive  behaviors 

management  provides  basic  needs  of  office  facilities  fairly  well 

office  facilities  are  a  drawback  to  working  in  the  program 

X 

working  conditions  are  fairly  comfortable,  time  off  policy  fairly  good 

working  conditions  and  time  off  policy  is  inconsistent  and  difficult  at  times 

X 

Communication: 


communications  primarily  written  (email) 

X 

communications  primarily  verbal  (face-to-face) 

detailed  instructions:  oral  presentation,  follow-up  email 

email  only 

X 

formal  communication  protocol 

informal  communications 

X 

external  vertical  communications  restricted 

X 

external  vertical  communication  allowed 

coders  notebook  weekly  accomplishment  reports  required 

not  required 

X 

user-coder  relationship  established,  encouraged,  and  mediated 

user-coder  interaction  minimized 

X 

meetings  structured  to  minimize  waster  time 

meetings  unstructured  and  open  ended 

X 

meetings  have  agenda,  objectives,  and  conclude  with  action  items 

meeting  agenda  fluid  and  open  ended 

X 

program  management  and  coder  communication  face  to  face 

program  management  and  coder  communication  primarily  email 

X 

program  team  updated  regularly  regarding  organizational  &  program  status 

meetings  infrequently  scheduled 

X 

open  communications  is  encouraged 

communication  hrough  chain  of  command  only  is  encouraged 

X 

program  manager  accessible  for  discussions 

program  manager  difficult  to  get  an  appointment  to  see 

X 

program  management  (PM)  is  viewed  as  separate  from  team 

X 

PM  mixes  with  team  frequently 

management  regularly  holds  team  meetings 

X 

meetings  are  sporadic 

meetings  are  structured  with  definite  goals  and  objectives 

X 

meetings  are  informal 

program  management  is  generally  easy  to  reach  and  talk  to 

PM  is  usually  hard  to  get  a  hold  of  and  difficult  to  talk  to 

X 

team-program  manager  relationship  adult-adult 

team-program  management  relationship  parent-child 

X 

schedules  are  spontaneous  and  poorly  communicated 

schedules  must  be  fixed  and  rigidly  followed  and  formally  reported 

X 

work  is  seen  as  complex  processes  involving  team  working  together 

work  broken  into  pieces  with  minimal  team  member  interaction 

X 

action  items  often  is  poorly  disseminated  and  usually  not  followed  through 

X 

action  items  communicated  and  followed  through  thoroughly 

team  members  require  frequent  clarifications  by  PM  for  assigned  tasks 

X 

team  members  rarly  require  clarifications  by  PM  for  assigned  tasks 

Pair  choice  section  THREE:  (People  Management)  choose  most  applicable  term  of  the  two  for  each  row  (page  2  of  2): 
Leadership: 


long  range  organizational  vision 

X 

short  tern  program  and  immediate  work  focus 

X 

X 

X 

X 

X 

X 

lead  through  personal  attention  to  others 

X 

action-oriented  leadership  approach 

run  as  much  of  the  organization  as  possible 

X 

let  team  make  decisions  as  much  as  possible 

direct  and  domineering  style 

X 

encourage  independence  in  others 

traditional  leaders  respect  hierarchy 

X 

do  what  needs  to  be  done 

win  cooperation  rather  than  demand  it 

tough-minded  with  others 

act  strongly  and  forcefully  in  the  field  of  ideas 

X 

prefer  to  lead  other  independent  types  while  seeking  autonomy  for  self 

consults  with  team  members  to  find  solutions  to  problems 

X 

consults  team  members  to  get  validation  of  PM's  predetermined  solutions 

keep  people  well  informed 

only  as  much  knowledge  as  necessary  for  their  work 

make  things  happen  by  focusing  on  the  immediate  problems 

X 

long  range  focus  and  de-emphasize  current  problems 

manage  others  loosely  and  prefer  minimal  supervision 

follow  traditional  procedures  and  rules  conscientiously 

leadership,  management  decisions  exclusively  by  program  management 

X 

program  management  makes  decisions  but  gets  inputs  from  team 

team-program  manager  relationship  adult-adult 

team-program  management  relationship  parent-child 

program  management  makes  decisions  but  gets  inputs  from  team 

X 

all  program  team  members  responsible  for  program  decisions 

when  a  problem  arises:  management  takes  over  to  solve  it 

X 

management  lets  the  team  solve  the  problems 

leadership  is  do  as  1  say,  not  do  as  1  do 

X 

leadership  by  example 

program  expectation  not  influenced  by  PM 

X 

program  expectation  managed  by  PM 

PM  gives  freedom  to  team,  but  has  no  mentoring  for  members  (abdication) 

X 

PM  empowers  teams  by  mentoring  members  to  be  leaders 

promgram  management  waits  and  sees  what  happens  then  plans 

X 

management  plans  far  in  advance 

program  management  is  constantly  reacting  to  emergencies 

X 

management  is  one  step  ahead  of  problems 

facilitative  approach  to  solving  problems 

X 

take  charge  readily  and  often 

program  management  is  complex,  takes  much  time  to  understand 

management  is  simple,  easy  to  figure  out 

program  management  prefers  to  plunge  right  in 

X 

takes  time  to  separate  things  to  be  done  and  order  of  doing  them 

program  management  reacts  spur  of  the  moment 

X 

methodically  follows  plans 

Technical  Competency  of  the  Program  Manager: 

PM  has  technical  experience  particular  to  the  particular  s/w  program 

X 

PM  relies  on  team  members  solely 

X 

X 

X 

X 

X 

PM  participates  in  technical  reviews 

X 

PM  only  in  non-technical  reviews 

PM  participates  in  making  technical  decisions  when  problems  arise 

X 

PM  delegates  technical  questions 

PM  does  not  get  involved  discussing  technical  options 

X 

PM  contributes  to  technical  options  being  discussed 

PM  does  not  review  technical  options  and  decisions 

X 

PM  reviews  technical  options  and  decisions 

PM  actively  attempts  to  keep  up-to-date  with  current  technology  and  standards 

PM  is  removed  from  cutting  edge  technology  issues 

PM  receives  technical  periodicals  and  occasionally  references  applicable  articles 

PM  doesn't  read  periodicals  nor  reference  current  articles  to  team 

PM  doesn't  have  technical  background  (or  education) 

PM  has  technical  background  (or  education) 

team  members  avoid  PM  when  they  need  technical  advice 

team  members  generally  consider  talking  to  PM  regarding  technical  issues 

HR  [2]  +  Comm.  [4]  +  Leadership  [6]  +  Tech.  Competency  [6]  =  People  Mgmt.  score  [18]  Enter  on  QMM  scoresheet  blk  c. 
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Pair  choice  section  FOUR:  (Risk  Management(RM))  choose  most  applicable  term  of  the  two  for  each  row  (page  1  of  2): 


RM  is  formal  and  documented 

X 

RM  is  informal,  if  at  all 

a  risk  management  plan  exists 

X 

no  risk  management  plan  is  developed 

RM  is  more  of  a  data  call  than  a  useful  document 

RM  drives  decisions  on  the  program 

RM  is  done  prior  to  the  program  beginning 

RM  is  done  prior  and  during  program  execution 

RM  is  only  done  during  the  program  execution 

RM  is  done  prior  and  during  program  execution 

risks  are  generalized  through  the  whole  program 

X 

risks  are  categorized 

risk  management  is  done  internally,  only 

an  outside  organization  also  contributes  to  the  RM  process 

risk  is  a  management  function 

risk  is  a  program  team  function 

risks  are  precisely  articulated 

X 

risks  are  generalized,  if  at  all 

each  risk  has  a  consequence 

X 

consequences  are  generalized,  if  at  all 

a  mitigation  strategy  is  completed  for  each  risk 

X 

mitigation  strategy  is  generalized,  if  at  all 

contingency  plans  are  developed  for  a  RM  plan 

X 

contingency  plans  are  ad  hoc  as  problems  arise  in  the  program 

risks  are  anticipated 

if  problems  arise,  management  will  deal  with  it 

the  program  doesn't  have  any  risk 

programs  that  do  not  have  risk,  have  problems 

risk  management  is  automated 

risk  management  may  use  tools,  but  depend  on  human  input 

risks  are  assigned  probabilities 

X 

probabilities  are  not  relevant  for  RM 

all  risks  are  potential  problems,  relative  priorities  for  risks  are  not  useful 

risks  are  weighed  relative  to  other  program  risks  and  thus  prioritized 

risk  management  information  is  only  shared  internally 

risk  management  information  is  shared  with  all  stakeholders 

risk  analysis  uses  ordinal  rankings 

risk  analysis  uses  actual  measurements  with  a  mathematical  model 

regret  analysis  used 

X 

no  regret  analysis  done 

attach  probabilities  to  future  events 

X 

no  probabilities  associated  with  future  events 

assessing  risks  with  mechanical  meethods 

risks  should  be  compared  to  other  risks  and  sorted 

risk  status  tracked 

X 

not  tracked 

technical  risks  examined 

X 

no  technical  risks  examined 

process  risks  examined 

X 

no  process  risks  examined 

product  risks  examined 

X 

no  product  risks  examined 

stakeholder/user  risks  examined 

X 

no  examination  of  stakeholder/user  risks 

checklists  used  to  identify  risks 

X 

no  checklists  used 

risks  are  tracked 

X 

no  tracking  or  monitoring  of  risks 

each  risk  has  an  impact 

X 

no  impact  analysis  of  risk 

each  risk  has  a  mitigation  plan 

X 

no  individual  risk  mitigation 

risks  monitored  by  priority 

X 

no  special  attention  to  track  higher  priority  risks 

risk  assessment  is  formalized 

X 

no  formal  risk  assessment 

risk  control  is  formalized 

X 

no  formal  risk  control 

integration  risks  not  considered 

integration  risks  examined 

Risk  Management  (page  1  of  2)  score  j 

1  33  1 
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Pair  choice  section  FOUR:  (Risk  Management(RM))  choose  most  applicable  term  of  the  two  for  each  row  (page  2  of  2): 


risks  to  cost 

X 

no  cost  risks  examined 

unforeseen  risks  have  occurred  in  program 

any  risk  that  came  up  had  been  identified  previously 

personnel  risks  examined 

X 

no  personnel  risks  examined 

estimation  risks  examined 

X 

no  estimation  risks  examined 

planning  risks  examined 

X 

no  planning  risks  examined 

requirements  risks  examined 

X 

no  requirements  risks  examined 

resource  risks  examined 

X 

no  resource  risks  examined 

risk  management  plan  updated  regularly 

X 

no  regular  risk  management  plan  updates 

risks  charted 

X 

risks  not  charted 

performance  risks  examined 

X 

performance  risks  not  examined 

program  management  self  risks  examined 

X 

no  program  management  risks  examined 

risk  from  program  constraints  examined 

X 

no  program  constraint  risks  examined 

each  category  of  risks  are  prioritized 

X 

no  prioritization 

each  category  of  risks  are  evaluated  for  impact 

X 

no  impact  analysis  performed 

each  category  of  risks  have  control  strategy 

X 

no  control  strategy 

documentation  risks  examined 

X 

no  documentation  risks  examined 

regret  matrix  tracked 

X 

no  regret  matrix  or  not  tracked 

communication  of  risk  activities  are  facilitated 

X 

no  facilitation  or  promotion  of  communication  of  risk  activities 

taxonomy-based  questionnaire  used  to  identify  risks 

X 

taxonomy-based  questionnaire  not  used 

associated  hardware  risks  examined 

X 

no  consideration  for  hardware  risks 

integration  risks  examined 

X 

integration  risks  not  examined 

communication  risks  examined 

X 

communication  risks  not  examined 

leadership  risks  examined 

X 

leadership  risks  not  considered 

risk  avoidance  considered  for  certain  risks 

X 

risk  avoidance  not  considered  for  risks 

risk  documentation  forms  used 

X 

no  risk  documentation  forms  used 

dependency  risks  examined 

X 

no  dependency  risks  examined 

alternatives  like  risk  avoidance  considered  for  high  risk  items 

X 

no  consideration  of  risk  avoidance 

documented  risk  statements  use  a  condition-consequence  type  format 

X 

condition-consequence  of  risk  statements  not  clearly  defined 

no  assignment  of  ownership  of  risk  mitigation  action 

X 

each  risk  mitigation  action  is  assigned  to  an  individual  for  resolution 

calculation  of  risk  exposure  made  (probability  X  loss,  for  each  risk) 

no  risk  exposure  calculations 

oral  communication  of  risks  only 

X 

risks  written  in  a  way  that  communicates  nature  and  status  of  factors 

triggers  used  to  quantify  risk  conditions  present 

risk  conditions  present  are  all  subjective 

risk  "czar"  in  program  for  monitoring  risks 

no  special  positions/responsibilities  for  risk  monitoring 

post-program  review  completed  (scheduled)  for  unanticipated  problems  ID 

no  post-program  reviews  completed  or  scheduled 

no  schedule  risks  examined 

risks  to  schedule  investigated 

Risk  Management  (pg  2  of  2)  score  [29]  +pg  1  score  [33]  =  TOTAL  SCORE  [62]  Enter  on  QMM  scoresheet  blk  d. 
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F.  PROGRAM  C  -  ASSOCIATE 

1.  QMM  Summary  Score  Sheet 


QMM  Scoresheet 

Part  One 

Part  Two 

Total 

Importance 

Weighted 

Category 

Score 

Score 

Score 

Coefficient 

Score 

Requirements 

Management 

a 

45 

e 

5 

50 

0.92 

= 

46 

Est./Planning 

Management 

D 

54 

D 

38 

92 

0.67 

= 

61.64 

People 

Management 

c 

16 

g 

-15 

1 

1.86 

= 

1.86 

Risk  Management 

D 

61 

D 

46 

107 

0.55 

= 

58.85 

QMM  SCORE  168.35 


Max.  QMM  score  possible  528.00 

Min.  QMM  score  possible  -130.86 

QMM  percentage  score:  45.41% 


Objective/Subjective  view  of  the  overall  success  of  program  A  on  a  scale  of  0  to  10 
(0  being  total  failure,  10  being  perfect  program  total  success) 

Survey  Participant:  Associate 

Success  Score:  6 


100 


2.  Requirements  Management  Questionnaire  Responses 


No.  Requirements  Management  Questionnaire  -  Total:  Block  e _ Yes  No  N/A 


1 

PM  chose  to  have  a  formal  requirements  list 

X 

2 

Requirements  recorded  in  some  way 

X 

3 

Written  requirements  were  part  of  some  formal  document 

X 

4 

Written  requirements  were  informal 

X 

5 

At  least  some  requirements  were  oral  only 

X 

6 

All  stakeholders  were  identified 

X 

7 

All  stakeholders  participated  in  the  requirements  extraction 

X 

8 

Some  stakeholders  participated  in  the  requirements  extraction 

X 

9 

Management  extracted  requirements,  no  stakeholder  involvement 

X 

10 

Management  passed  requirements  to  development  team 

X 

11 

Stakeholders  not  involvved  in  Management  extraction,  but  approved 

X 

12 

Management  gets  inputs  from  stakeholders,  then  develops  requirements 

X 

13 

Developers  work  informally  with  users  to  arrive  at  requirements 

X 

14 

Same  as  13,  but  management  oversees  and  formalizes 

X 

If  a  waterfall  or  sequential  development  strategy:  1 

15 

All  requirements  complete  before  design 

16 

Some  requirements  left  incomplete  prior  to  design 

17 

Requirements  informal  prior  to  design  effort 

18 

Requirements  serve  as  input 

19 

Length  of  time  for  requirements  work  greater  than  development  work 

20 

Requirements  developed  in  parallel  to  design 

OR  If  a  prototype,  throwaway,  or  other  development  strategy:  | 

15 

Learn  about  requirements  through  development  efforts 

X 

16 

No  coding  until  all  requirements  are  defined 

X 

17 

Requirements  formal  prior  to  design  effort 

X 

18 

Requirements  serve  as  output 

X 

19 

Requirements  definition  work  in  parallel  to  development  efforts 

X 

20 

Requirements  developed  in  parallel  to  design 

X 

21 

Are  requirements  frozen  at  some  phase 

X 

22 

Change  management  exists 

X 

23 

Change  management  is  formal 

X 

24 

Project  strategy  is  consistent  throughout  development 

X 

25 

Requirements  are  updated 

X 

26 

Configuration  Management  (CM)  exists 

X 

27 

CM  is  formal 

X 

28 

Requirements  are  testable 

X 

29 

Requirements  testing  considered/implemented  during  extraction 

X 

30 

Requirements  testing  plan  exists 

X 

31 

Requirements  testing  is  formal 

X 

32 

All  requirements  have  priorities 

X 

33 

All  requirements  must  be  implemented 

X 

34 

Requirements  are  tested 

X 

35 

All  requirements  are  equally  important 

X 

36 

At  least  some  requirements  have  priorities 

X 

37 

All  requirements  are  traceable 

X 

38 

Traceability  not  important 

X 

39 

Each  requirement  has  an  author 

X 

40 

Who  authored  requirement  is  not  important 

X 

41 

Initial  set  of  requirements  to  be  implemented,  no  requirements  creep 

X 

42 

Structured  and  tracked  changes  to  requirements  only 

X 

43 

Change  is  inevitable,  changes  allowed  at  all  times 

X 

44 

Change  is  inevitable,  but  changes  limited 

X 

45 

Requirements  control  funding 

X 

46 

Requirements  history  kept 

X 

47 

Baseline  established  for  requirements  at  some  point  prior  to  develop 

X 

TOTAL  SCORING 
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3.  Estimation/Planning  Questionnaire  Responses 


No.  Estimation/Planning  Questionnaire  -  Total:  Block  f 

Yes 

No 

N/A 

1 

A  volume  product  metric  used  (LOC,  #  of  files,  #of  screens,  pages  of  doc) 

X 

2 

Measure  used  for  various  product  elements  (modules,  components,  CSCI) 

X 

3 

Product  measures  made  by  phase  (amt  at  implementation,  LOC  changed  at  unit  test) 

X 

4 

Other  product  attributes  measured  (FP,  throughput,  mem  cap,  cyclomatic  complexity) 

X 

5 

Product  matrics  tracked  and  updated  hroughout  program  execution 

X 

6 

Event  count  process  metric  used  (#  defects  in  test,  reqmt  changes,  milestones  met) 

X 

7 

Time  measure  process  metric  used  (cycle  time) 

X 

8 

Process  metrics  tracked  and  updated  throughout  program  execution 

9 

Program  cost  estimations  made  from  product  or  process  metrics 

X 

10 

Program  cost  extirpations  tracked  and  updated  to  reflect  progress/changes 

X 

11 

Factor  analysis  performed  on  program 

X 

12 

Program's  primary  purpose,  including  major  functions  and  deliverables  known 

13 

Work  breakdown  structure  developed 

X 

14 

Task  estimated  with  realistic  expectations  of  productivity  probabilities 

X 

15 

Schedules  developed  based  on  realistic  expectations 

X 

16 

Schedules  tracked  and  updated  based  on  new  information 

X 

17 

Detailed  activity  lists  used  for  clearly  defined  completed/not  completed  tasks 

18 

Quality  assurance  plan  orsimilarto  aid  in  detecting  defects  early  in  program 

X 

19 

COCOMO  estimates  performed 

X 

20 

CSCI  clearlydefined  and  tasked 

21 

Estimates  completed  ad  hoc 

X 

22 

Gantt  charts  used  and  updated 

X 

23 

Resource  estimations  (working  hrs,  job  categories,  task  activities)  done 

X 

24 

Earned  value  established 

X 

25 

Earned  value  tracked  throughout  program 

X 

26 

Quality  expectations  established  for  product  with  users  and  stakeholders 

X 

27 

Critical  path  for  program  tasks  developed  and  tracked 

X 

28 

Measure  of  effectiveness  (MOE)  or  Figure  of  merit  established  and  tracked 

X 

29 

Estimates  are  updated  routinely 

X 

30 

Schedules  are  updated  routinely 

X 

31 

Estimations  are  made  by  program  management  (top-down) 

X 

32 

Estimateions  are  made  by  program  team  members  (bottom-up) 

X 

33 

Automated  program  tracking  used 

X 

34 

PM  usuallythorough  in  tracking  and  reporting  schedules  and  financials 

35 

WBS  developed  only  as  data  call 

36 

Earned  value  used  to  track  program  progress 

X 

37 

PM  insists  on  prioritizing  work  reduction  as  schedule/funding  compromised  by  stakeholders 

~>r 

38 

Estimations  are  done  using  both  top  down  and  bottoms  up  approaches 

39 

All  program  team  members  involved  in  planning  process 

40 

Hardware  also  considered  in  estimaation  process 

~>r 

41 

Program  history  compiled 

X 

42 

System  upgrades  (SCR)  software  changes  requests  estimated  individually 

43 

Management  duties  apart  of  each  team  member's  responsibilities 

44 

PM  dictates  schedules  to  program  team 

X 

45 

Code  reviews  planned  in  schedule 

X 

46 

Defined  tangible  milestones  established  for  program  tasks 

X 

47 

Test  planning  done  at  the  start  of  the  program 

X 

48 

Estimations  are  completed  bythose  performing  the  tasks 

49 

Sensitivityanalysis  performed  for  program  choices 

50 

Software  deployment  planning  completed 

~ 

TOTAL  SCORING 

38 
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4.  People  Management  Questionnaire  Responses 


No.  People  Management  Questionnaire  -  Total:  Block  g _ Yes  No  N/A 


1 

PM  is  accessible  in  person  by  each  team  member 

X 

2 

PM  is  accessible  via  email  (memo,  letter)  by  each  team  member 

X 

3 

PM  is  accessible  via  phone  by  each  team  member 

X 

4 

PM  not  only  considers  a  person's  suitability,  not  also  desire  to  be  on  a  team 

X 

5 

PM  consults  with  each  team  member  regarding  their  career  goals 

X 

6 

PM  regularly  holds  meetings  to  inform  team  of  program  progress 

X 

7 

PM  solicits  opinions  from  team  members  before  making  decisions 

X 

8 

PM  lets  teams  make  decisions  affecting  their  work 

X 

9 

PM  freuently  makes  decisions  without  any  consultation  with  members 

X 

10 

PM  understands  the  technology/language  of  the  program 

X 

11 

PM  is  able  to  communicate  with  other  the  technical  issues  in  the  program 

X 

12 

PM  prioritized  problems  or  conflicts  within  the  program 

X 

13 

PM  assists  team  members  in  developing/advising  of  career  path 

X 

14 

PM  empowers  program  members  to  recommend  hiring  new  team  members 

X 

15 

PM  empowers  program  members  to  recommend  firings  of  other  members 

X 

16 

PM  specifically  assigns  work  to  each  program  member 

X 

17 

PM  sets  communication  protocol 

X 

18 

PM  allows  unrestricted  communications 

X 

19 

PM  encourages  or  requires  training  for  each  individual 

X 

20 

PM  takes  control  in  difficult/roblem  areas 

X 

21 

PM  looks  ahead  to  new  programs,  new  upgrades  of  existing  program 

X 

22 

PM  maintains  regular  communications  with  all  stakeholders 

X 

23 

PM  maintains  regular  communications  with  users 

X 

24 

PM  encourages  program  team  communication  with  users 

X 

25 

PM  encourages  program  team  communication  with  stakeholders 

X 

26 

PM  facilitates  horizontal  communication  within  program 

X 

27 

PM  facilitates  communication  during  integration 

X 

28 

PM  holds  meetings  without  clear  objectives 

X 

29 

PM  must  approve  all  decisions  within  the  program 

X 

30 

PM  must  approve  all  interactions  with  stakeholders 

X 

31 

PM  must  approve  all  interactions  with  users 

X 

32 

PM  makes  all  presentations  to  stakeholders/users 

X 

33 

PM  is  considered  "flexible"  in  terms  of  program  members  personal  issues 

X 

34 

PM,  at  least  occasionally,  schedules/promotes  outside  work  team  activities 

X 

35 

PM  is  readily  willing  to  listen  to  program  prblems  and  complaints 

X 

36 

PM  takes  action  to  resolve  program  problems  and  complaints 

X 

37 

PM  is  generally  respected  by  stakeholders,  users,  and  organization 

38 

PM  sometimes  fails  to  grasp  important  technical  issues  in  program 

X 

39 

PM  recruits  program  team  members  from  outside  organization 

X 

40 

PM  participates  in  technical  reviews 

X 

41 

Program  personnel  have  clearly  defined  specific  tasks 

X 

42 

Although  individual's  tasks  are  specific,  each  exposed  to  the  "bigger  picture" 

X 

43 

PM  has  clearly  defined  his/her  expectations  for  each  individual 

X 

44 

PM  delegation  of  duties  is  usually  seemless  in  execution 

X 

45 

PM  acts  as  facilitator  to  solving  personnel  conflicts 

X 

46 

PM  attempts  to  motivate  individuals  on  the  program  team 

X 

47 

PM  clearly  spearates  technical  from  managerial  roles  for  individuals 

X 

48 

PM  directs  how  he/she  expects  the  task  to  be  accomplished 

X 

49 

PM  directs  what  needs  to  be  done,  but  does  not  direct  how 

X 

50 

PM  attempts  to  spotlight  individuals  in  the  program  for  positive  exposure 

X 

TOTAL  SCORING 
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5. 


Risk  Management  Questionnaire  Responses 


No.  Risk  Management  Questionnaire  -  Total:  Block  h 

Yes 

No 

N/A 

1 

Risk  Management  (RM)  is  specifically  an  activity  in  the  program 

X 

2 

RMis  formal  and  documented 

X 

3 

Aspecific  RM  Ian  exists 

X 

4 

RM  is  required  in  the  program,  but  not  used  during  the  program 

X 

5 

RM  is  done  prior  to  the  program  execution 

X 

6 

RM  is  done  by  an  outside  entity  to  the  development 

X 

7 

RMis  done  internally  only 

X 

8 

RMis  both  internally  performed  and  externally  assessed 

X 

9 

RM  planning  occurs  during  or  after  major  milestones  in  the  program 

X 

10 

Risk  Assessment  is  only  a  management  function 

X 

11 

RM  is  informal  or  non  existent 

X 

12 

There  is  a  RM  plan,  but  it  is  not  updated  or  tracked 

X 

13 

Risks  are  onlygeneralized 

X 

14 

Each  risk  is  delineated 

X 

X 

15 

Each  risk  has  a  consequence 

X 

X 

16 

Each  risk  has  a  likelihood  rating  of  some  sort 

X 

17 

Each  risk  has  a  mitigation  strategy 

X 

18 

Risk  Management  is  automated 

X 

19 

Risks  are  tracked 

X 

20 

21 

Regret  analysis  performed 

X 

22 

RM  drives  decisions  in  the  program 

X 

23 

Risks  have  probabilities 

X 

24 

Risk  Management  is  ad  hoc 

X 

25 

RM  information  is  shared  with  all  stakeholders  (as  appropriate) 

X 

26 

Risks  are  weighed  relative  to  other  program  risks 

X 

27 

Risk  Assessment  is  a  program  team  activity 

X 

28 

Risk  Assessment  done  prior  to  program  start 

X 

29 

Risk  Assessment  includes  personnal  risk 

X 

30 

RMuses  tools,  but  depends  on  human  decisions 

X 

31 

Risk  assessment  includes  cost  risks 

X 

32 

Risk  Assessment  includes  schedule  risks 

X 

33 

Risk  Assessment  includes  technology  risks 

X 

34 

Risk  Assessment  is  briefed  organization  structure  above  program  manager 

X 

35 

Risk  Assessment  includes  requirements  risks 

X 

36 

Risk  Assessment  includes  user  risks  (too  little  involvement  of  user) 

X 

37 

Risk  Assessment  includes  documentation  risks 

X 

38 

Risk  Assessment  includes  integration  risks 

X 

39 

Risk  Assessment  includes  interface  risks  (non-standard) 

X 

40 

Risk  Assessment  includes  continuing  requirements  change  (feature  creep) 

X 

41 

Risk  Assessment  includes  dependent  projects/programs  risks 

X 

42 

Documentation  proof  exists  to  demonstrate  following  risk  management  plan 

X 

43 

High  rish  have  measured  tracking  (high  profile  status) 

X 

44 

Organizational  history  used  to  search  for  risks 

X 

45 

Other  organizational  checklists  used  for  risk  assessment 

X 

46 

Internal  organizational  checklists  used  for  risk  assessment 

X 

47 

Risk  Assessment  information  contributed  to  internal  or  other  database 

X 

48 

Risk  Assessment  includes  internal  organization  risks 

X 

49 

Risk  Assessment  includes  stakeholder  risks 

X 

50 

No  risk  management  needed;  program  is  straightforward  &  understood 

X 

TOTAL  SCORING 
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6.  Pair  Choices  Responses 

Pair  choice  section  ONE:  (Requirements  Management)  choose  most  applicable  term  of  the  two  for  each  row  (page  1  of  2): 


formal  requirement  list 

X 

informal  requirement  list 

w  ritten  requirements 

X 

oral  requirements 

requirements  informal,  but  recorded 

X 

requirements  not  recorded 

requirements  as  part  of  an  SRS  (or  other  formal  repository) 

X 

requirements  informally  recorded 

requirements  taken  as  is  from  customer 

look  to  reformulate,  interview  in-depth,  or  otherwise  re-validate 

only  one  development  strategy  used 

X 

strategies  not  consistent,  used  at  different  times 

stakeholders  as  part  of  requirements  development 

X 

stakeholders  approving  requirements  after  formulated  by  development  team 

requirements  are  testable 

X 

requirements  have  no  test  plans 

informal  test  plan  or  no  test  plan 

formal  test  plan 

test  team  involved  w  ith  requirements 

X 

no  test  team  input  or  plans  during  requirements  development 

only  a  percentage  of  requirements  present  in  baseline 

baseline  must  contain  all  requirements 

requirements  documentation  has  hierarchical  structure 

all  requirements  must  be  implemented 

requirements  have  listed  responsible  party 

requirements  origin  not  important 

requirements  documentation  have  versions 

no  requirements  history 

requirements  have  specific  attribute  values 

requirements  all  rank  evenly 

funding  controls  requirements  definition 

requirements  definition  controls  funding 

reqquirements  are  top  dow  n 

X 

requirements  are  bottom  up 

users/stakeholders  are  identified  and  interviewed  (market  survey) 

X 

no  special  consideration  to  identify  users/stakeholders 

each  requirement  has  a  singular  concept 

some  requirements  are  compound  statements 

requirements  definition  minimized  when  funding  short 

X 

program  scope  may  reduce,  but  requirements  definition  completed 

requirements  extraction  has  formal  process 

X 

requirements  extraction  ad  hoc 

change  procedures  formal 

X 

change  procedures  ad  hoc 

users/stakeholders  somehow  involved  in  requirements  definition 

program  team  only  involved  in  requirement  definition 

management  sets  requirements  for  developers 

X 

developers  at  least  partially  involved  in  setting  requirements 

requirements  changed  at  least  once  since  baseline  established  prior  to  newversion 

X 

requirements  in  baseline  has  not  changed  prior  to  newversion  or  upgrade 

no  ranking  of  requirements 

X 

requirements  have  priorities  assigned 

use-case  diagrams  (or  other  models  or  scenario  developments) 

no  models  used  for  requirements  extraction 

requirements  changes  informal 

requirements  changes  formal 

plan  to  "freeze"  requirements  at  some  designated  milestone 

no  provision  for  "freezing"  requirements 

requirements  must  be  traceable 

X 

origin  of  requirements  not  important 

requirements  must  be  testable 

X 

system  developed  must  be  testable 

test  plans  to  determine  requirements  implemented 

no  test  plans  needed  for  requirements  verification 

requirements  have  priorities  in  implementation 

all  requirements  must  be  implemented 

some  requirements  have  multiple  statements  or  ideas 

one  idea,  one  statement  per  requirement 

Requirements  Management  (page  1  of  2)  score  |  29 
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Pair  choice  section  ONE:  (Requirements  Management)  choose  most  applicable  term  of  the  two  for  each  row  (page  2  of  2): 


j  ANSWER  THIS  BLOCK  OF  QUESTIONS  ONLY  IF  A  SEQUENTIAL  OR  WA  TERFALL  APPROACH  IS  USED  FOR  DEVELOPMENT  (Requirements  page  2  of  2)  | 

requirements  first,  then  initial  development  work 

initial  development  work  then  requirements 

requirements  documentation  driving  development 

requirements  documentation  developed  in  parallel/after  development 

user  feedback  considered  during  development 

after  development  starts,  user  feedback  serves  as  input  to  new  work 

change  management  procedures  used  strictly 

change  management  procedures  as  guidance  only 

design  decisions  prior  to  or  in  parallel  to  requirrements  development 

design  decisions  only  after  approved  requirements  stabilized 

requirements  summarized  wht  we  have  developed 

requirements  are  the  blueprint  for  development 

length  of  time  for  requirements  work  greater  than  development  work 

length  of  time  for  requirements  work  less  than  development  work 

requirements  have  design  detail 

no  design  detail  in  requirements 

requirements  creep  to  be  avoided 

requirements  creep  o.k.,  but  need  to  be  controlled 

freeze  requirements  at  some  point 

requirements  are  fluid  throughout  development 

formal  change  procedure 

informal  change  procedure 

change  management  plan 

no  change  management  plan 

requirements  ambiguity  always  present  to  some  extent 

requirements  ambuiguity  unacceptable  at  any  level 

testing  considered  up  frornt  during  requirements  determination 

testing  considered  down  the  line  during  development 

requirements  development  team  members  different  from  implementation 

those  working  on  requirements,  work  on  implementation 

start  implementation  as  early  as  possible  to  help  define  requirements 

requirements  must  be  defined  prior  to  any  implementation  work 

i|  ANSWER  THIS  BLOCK  OF  QUESTIONS  ONLY  IF  A  PROTOTYPING,  THROWAWAY,  SYNCHRONIZE  &  STABILIZE,  OR  OTHER  STRATEGY  USED  | 

develop  prototype,  then  determine  requirements 

X 

determine  requirements  prior  to  any  development  work 

requirements  testing  done  after  each  iteration 

X 

no  testing 

individual  changes  as  necessary 

X 

only  block  changes  made 

development  team  decides  on  changes  after  iteration 

users  involved  with  changes 

X 

changes  based  on  feedback  only  from  user  for  correction  of  problems 

X 

changes  to  upgrade  system  and  correct  problems 

funding  controls  changes  and  change  procedures 

X 

changes  control  funding 

requirements  documentation  finalized  prior  to  development 

X 

requirements  fluid  throughout  development  (only  freeze  at  end) 

requirements  test  plans  completed  prior  to  development 

X 

requirements  test  plans  completed  after  development 

requirements  first,  then  initial  development  work 

X 

initial  development  work  then  requirements 

use  development  effort  to  learn  more  about  requirements 

X 

define  all  requirements  prior  to  coding  anything 

requirements  ambiguity  always  present  to  some  extent 

X 

requirements  ambiguity  unacceptable  at  any  level 

requirements  have  design  detail 

X 

no  design  detail  in  requirements 

user  feedback  considered  during  development 

X 

after  development  starts,  user  feedback  serves  as  input  to  new  work 

get  something  to  users  as  soon  as  possible  for  evaluation 

X 

make  sure  it  is  complete  before  releasing 

management  dictates  requirements 

development  team  visually  represent  requirements  through  rapid  prototyping 

X 

new  requirements  allowed  after  initial  requirements  defined 

X 

new  requirements  not  allowed 

Requirements  Management  (pg  2  of  2)  score  [16]  +pg  1  score  [29]  =  TOTAL  SCORE  [45]  Enter  on  QMM  scoresheet  blk  a. 
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Pair  choice  section  TWO:  (Estimation/Planning  Management)  choose  most  applicable  term  of  the  two  for  each  row  (page  1  of  2): 


at  least  one  estimation  method  used  in  program 

X 

no  estimates 

formal  derivation  of  product  metric  for  estimation  of  size 

X 

ad  hoc  size  estimation 

ad  hoc  process  evaluation 

X 

formal  derivation  of  at  lest  one  process  metric 

develop  work  breakdown  structure  (WBS) 

X 

assign  work  as  needs  arise 

estimates  are  developed  to  fulfill  a  data  call  only 

use  estimates  to  plan  program 

use  estimates  to  sell  program  only 

estimates  are  useful  to  the  project  tema  for  planning  purposes 

resource  evaluations  made  for  program 

X 

no  resource  evaluation  for  planning 

use  both  bottom  up  &  top  down  for  estimate,  use  one  stakeholders  like 

X 

use  both  bottom  up  &  top  down  and  evaluate  significant  differences 

estimates  made  and  not  updated 

estimates  updated  throughout  program 

resources  estimations  used  to  adjust  product  size  estimate 

X 

estimations  made  irregardless  of  resources  available 

estimations  made  to  fit  budget 

budget  made  from  estimations 

estimations  compromised  to  get  program 

X 

rather  risk  loss  of  program  than  compromise  confident  estimations 

cycle  time  estimations 

X 

no  cycle  time  estimations 

event  count  estimations 

X 

no  event  count  estimations 

lines  of  code  (LOC)  estimation 

X 

no  LOC  estimation 

function  pont  (FP)  estimation 

X 

no  FP  estimation 

estimates  by  algorithmic  methods 

X 

estimates  by  analogy 

expert  judgement  for  estimates 

X 

ad  hoc  estimates 

estimates  by  algorithmic  methods 

X 

ad  hoc  estimates 

expert  judgement  for  estimates 

X 

estimates  by  analogy 

ad  hoc  estimates 

X 

estimates  by  analogy 

bottom  up  estimates 

X 

expert  judgement 

top  down  estimates 

X 

expert  judgement 

ad  hoc  estimates 

X 

any  other  estimate  process 

fuzzy  logic  estimating  method 

X 

no  formal  estimation  methodology 

WBS  development  from  estimates 

X 

WBS  development  in  parallel  or  prior  to  estimation  completion 

critical  path  of  program  determined 

X 

tasks  developed  but  no  path  is  identified 

estimators  are  program  team  members 

X 

estimators  are  outside  program  team 

management  only  on  estimations 

X 

all  team  members  involved  in  estimation  process 

estimates  updated  at  reviews 

X 

no  updates  of  estimates 

estimates  updated  at  reviews 

X 

estimates  constantly  updates  (in  between  reviews,  to) 

estimate  procedures  stay  the  same 

X 

estimate  procedures  change 

stakeholders  are  part  of  estimation  process 

X 

stakeholders  brief  estimations  after  completion 

estimates  are  used  beyond  initial  selling  of  program 

X 

estimates  are  one  time  events,  used  for  a  specific  purpose  once 

WBS  has  objective  measure  of  completeness 

X 

important  to  have  WBS  as  guide,  not  rigid  implementation 

Estimation/Planning  Management  (page  1  of  2)  score  |  27  | 
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Pair  choice  section  TWO:  (Estimation/Planning  Management)  choose  most  applicable  term  of  the  two  for  each  row  (page  2  of  2): 


life  cycle  estimates 

X 

estimates  for  program  initiation  only 

system  upgrades  (SCR)  software  change  requests  estimated  individually 

X 

systems  upgrades  estimated  as  w  hole 

estimates  for  on-gong  resources  needed  to  maintain  s/w 

X 

estimates  for  maintenance  not  done 

informal  re-estimates  during  development 

formal  re-estimates  at  pre-defined  milestones 

formal  re-estimates  w  hen  amendment  changing  the  system  is  introduced 

X 

informal  re-estimates  w  hen  amendment  changing  the  system 

person  in-charge  of  estimation  w  alks  in  a  managers  office  to  get  an  opinion 

X 

meeting(s)  organized  for  purpose  of  performing  cost  estimations 

factor  analysis  prior  to  commencement  of  program 

X 

none  done 

change  control  procedures  set  in  place 

X 

no  set  procedures 

elapsed  time  and  actual  w  ork  time  estimates 

X 

one  or  the  other  or  neither 

no  schedule  created 

scheudle  created 

schedule  not  updated 

schedule  updated 

schedule  followed 

X 

schedule  not  followed 

tasks  identification  arises  as  program  progresses 

X 

detailed  level  tasks  identified  prior  to  program  initiation 

scope  of  program  understood  by  all 

X 

scope  not  explicitly  defined 

quality  factors  and  criteria  identified 

X 

no  explicit  quality  factors  defined 

no  project  tracking  tools  used 

X 

project  tracking  tools  used 

CSCIs  identified  and  tasked 

X 

CSCIs  not  explicitly  identified 

expectations  are  managed  via  estimations 

X 

estimations  are  made  to  fit  preconceived  expectations 

no  cost  schedule  developed 

cost  schedule  developed 

no  resource  schedule  developed 

resource  schedule  developed 

team  members,  management  know  at  any  time  if  in  budget  &  schedule 

X 

exact  budget  &  schedule  status  somew  hat  unclear  to  at  least  some 

individual  program  phases  are  estimated 

X 

only  top  level  program  estimated 

stakeholders/users  emphasis  understood-quick  to  field  or  all  complete 

X 

program  management  sets  delivery  tradeoffs  w  ithout  outside  input 

testing  planned  w  ith  initial  program  planning 

X 

testing  not  in  initial  planning 

documentation  not  considered  ininitial  planning 

X 

documentation  part  of  initial  planning 

hardware  considered  in  estimations 

X 

software  only  considered 

no  formal  schedule/cost  tracking 

formal  procedures  established  for  tracking  cost  and  schedule 

earned  value  set  up 

X 

earned  value  not  used 

estimations  omit  documentation  planning 

X 

documentation  in  estimates 

training  omitted  in  estimates 

X 

training  part  of  estimates 

earned  value  set  up,  but  not  tracked 

X 

earned  value  tracked 

detailed  planning  done  w  ith  incomplete  set  of  requirements 

X 

detailed  planning  done  w  ith  detailed  set  of  requirements 

complete  infrastructure  support  mechanism  understood  for  estimations 

X 

no  consideration  of  infrastructure  done  for  estimations 

team  possibilities  considered  for  planning  of  program 

X 

no  consideration  for  outside  teaming  possibilities 

w  ork  breakdow  n  structure  (WBS)  set  up 

X 

no  WBS  completed 

Estimation/Planning  Management  (pg  2  of  2)  score  [27]  +pg  1  score  [27]  =  TOTAL  SCORE  [54]  Enter  on  QMM  scoresheet  blk  b. 
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Pair  choice  section  THREE:  (People  Management)  choose  most  applicable  term  of  the  two  for  each  row  (page  1  of  2): 
Human  Resources 


program  team  members  have  clearly  deined,  segmented  roles 


work  responsibilities  are  shared 


formal  team  building  procedures  are  used 


no  formal  team  building  emphasized 


program  manager  flexible  regarding  work  hours 


program  manager  maintains  strict  standards  for  work  hours 


big  picture  conveyed  to  all  team  members  by  program  management 


program  management  focuses  on  the  partitioned  tasks  with  team 


people  issues  dealt  with  primarily  through  indirect  methods  (email,  memo,  etc) 


people  issues  dealt  with  primarily  through  direct  methods  (face-to-face) 


training  is  required  and  planned  on  a  regular  basis 


training  is  ad  hoc 


each  team  member  is  educated  on  and  understands  overall  program  and  their  roles 


team  members  only  know  their  respective  areas 


consideration  for  team  members'  career  goals  are  reflected  in  assignments 


team  members  must  adapt  to  tasks  that  are  assigned 


team  members  assignments  and  responsibilities  are  mostly  dictated  by  PM 


assignments  and  responsibilities  are  discussed  and  agreed  upon  with  PM 


management  leads  in  problem  solving 


management  facilitates  and  lets  team  lead  in  problem  solving 


management  welcomes  problems  as  challenges  and  opportunities 


management  views  problems  as  obstacles  and  grounds  for  punishment 


team  members  participate  in  performance  evaluations  of  peers 


Personnel  evaluations  are  strictly  PM  responsibility 


management  reinforcement  feedback  sparse  and  inconsistent,  if  any 


management  provides  timely  reinforcement  feedback  for  positive  behaviors 


management  provides  basic  needs  of  office  facilities  fairly  well 


office  facilities  are  a  drawback  to  working  in  the  program 


working  conditions  are  fairly  comfortable,  time  off  policy  fairly  good 


working  conditions  and  time  off  policy  is  inconsistent  and  difficult  at  times 


Communication: 


communications  primarily  written  (email) 


communications  primarily  verbal  (face-to-face) 


detailed  instructions:  oral  presentation,  follow-up  email 


email  only 


formal  communication  protocol 


informal  communications 


external  vertical  communications  restricted 


external  vertical  communication  allowed 


coders  notebook  weekly  accomplishment  reports  required 


not  required 


user-coder  relationship  established,  encouraged,  and  mediated 


user-coder  interaction  minimized 


meetings  structured  to  minimize  waster  time 


meetings  unstructured  and  open  ended 


meetings  have  agenda,  objectives,  and  conclude  with  action  items 


meeting  agenda  fluid  and  open  ended 


program  management  and  coder  communication  face  to  face 


program  management  and  coder  communication  primarily  email 


program  team  updated  regularly  regarding  organizational  &  program  status 


meetings  infrequently  scheduled 


open  communications  is  encouraged 


communication  hrough  chain  of  command  only  is  encouraged 


program  manager  accessible  for  discussions 


program  manager  difficult  to  get  an  appointment  to  see 


program  management  (PM)  is  viewed  as  separate  from  team 


PM  mixes  with  team  frequently 


management  regularly  holds  team  meetings 


meetings  are  sporadic 


meetings  are  structured  with  definite  goals  and  objectives 


meetings  are  informal 


program  management  is  generally  easy  to  reach  and  talk  to 


PM  is  usually  hard  to  get  a  hold  of  and  difficult  to  talk  to 


team-program  manager  relationship  adult-adult 


team-program  management  relationship  parent-child 


schedules  are  spontaneous  and  poorly  communicated 


schedules  must  be  fixed  and  rigidly  followed  and  formally  reported 


work  is  seen  as  complex  processes  involving  team  working  together 


work  broken  into  pieces  with  minimal  team  member  interaction 


action  items  often  is  poorly  disseminated  and  usually  not  followed  through 


action  items  communicated  and  followed  through  thoroughly 


team  members  require  frequent  clarifications  by  PM  for  assigned  tasks 


team  members  rarly  require  clarifications  by  PM  for  assigned  tasks 
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Pair  choice  section  THREE:  (People  Management)  choose  most  applicable  term  of  the  two  for  each  row  (page  2  of  2): 
Leadership: 


long  range  organizational  vision 

X 

short  tern  program  and  immediate  w  ork  focus 

lead  through  personal  attention  to  others 

action-oriented  leadership  approach 

X 

run  as  much  of  the  organization  as  possible 

X 

let  team  make  decisions  as  much  as  possible 

direct  and  domineering  style 

X 

encourage  independence  in  others 

traditional  leaders  respect  hierarchy 

X 

do  w  hat  needs  to  be  done 

w  in  cooperation  rather  than  demand  it 

X 

tough-minded  with  others 

act  strongly  and  forcefully  in  the  field  of  ideas 

X 

prefer  to  lead  other  independent  types  while  seeking  auto  no  my  fo  r  self 

consults  w  ith  team  members  to  find  solutions  to  problems 

X 

consults  team  members  to  get  validation  of  FMs  predetermined  solutions 

keep  people  well  informed 

X 

only  as  much  know  ledge  as  necessary  for  their  w  ork 

make  things  happen  by  focusing  on  the  immediate  problems 

long  range  focus  and  de-emphasize  current  problems 

X 

manage  others  loosely  and  prefer  minimal  supervision 

X 

follow  traditional  procedures  and  rules  conscientiously 

leadership,  management  decisions  exclusively  by  program  management 

X 

program  management  makes  decisions  but  gets  inputs  from  team 

team-program  manager  relationship  adult-adult 

X 

team-program  management  relationship  parent-child 

program  management  makes  decisions  but  gets  inputs  from  team 

X 

all  program  team  members  responsible  for  program  decisions 

w  hen  a  problem  arises:  management  takes  over  to  solve  it 

X 

management  lets  the  team  solve  the  problems 

leadership  is  do  as  1  say,  not  do  as  1  do 

X 

leadership  by  example 

program  expectation  not  influenced  by  PM 

X 

program  expectation  managed  by  PM 

PM  gives  freedom  to  team,  but  has  no  mentoring  for  members  (abdication) 

X 

PM  empow  ers  teams  by  mentoring  members  to  be  leaders 

promgram  management  w  aits  and  sees  w  hat  happens  then  plans 

management  plans  far  in  advance 

X 

program  management  is  constantly  reacting  to  emergencies 

X 

management  is  one  step  ahead  of  problems 

facilitative  approach  to  solving  problems 

X 

take  charge  readily  and  often 

program  management  is  complex,  takes  much  time  to  understand 

management  is  simple,  easy  to  figure  out 

X 

program  management  prefers  to  plunge  right  in 

X 

takes  time  to  separate  things  to  be  done  and  order  of  doing  them 

program  management  reacts  spur  of  the  moment 

X 

methodically  follows  plans 

Technical  Competency  of  the  Program  Manager: 


PM  has  technical  experience  particular  to  the  particular  s/w  program 

PM  relies  on  team  members  solely 

X 

PM  participates  in  technical  reviews 

X 

PM  only  in  non-technical  reviews 

PM  participates  in  making  technical  decisions  w  hen  problems  arise 

X 

PM  delegates  technical  questions 

PM  does  not  get  involved  discussing  technical  options 

X 

PM  contributes  to  technical  options  being  discussed 

PM  does  not  review  technical  options  and  decisions 

PM  reviews  technical  options  and  decisions 

X 

PM  actively  attempts  to  keep  up-to-date  with  current  technology  and  standards 

PM  is  removed  from  cutting  edge  technology  issues 

X 

PM  receives  technical  periodicals  and  occasionally  references  applicable  articles 

PM  doesn't  read  periodicals  nor  reference  current  articles  to  team 

X 

PM  doesn't  have  technical  background  (or  education) 

PM  has  technical  background  (or  education) 

X 

team  members  avoid  PM  w  hen  they  need  technical  advice 

X 

team  members  generally  consider  talking  to  PM  regarding  technical  issues 

HR  [4]  +  Comm.  [4]  +  Leadership  [4]  +  Tech.  Competency  [4]  =  People  Mgmt.  score  [16]  Enter  on  QMM  scoresheet  blk  c. 
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Pair  choice  section  FOUR:  (Risk  Management(RM))  choose  most  applicable  term  of  the  two  for  each  row  (page  1  of  2): 


RM  is  formal  and  documented 

X 

RM  is  informal,  if  at  all 

a  risk  management  plan  exists 

X 

no  risk  management  plan  is  developed 

RM  is  more  of  a  data  call  than  a  useful  document 

X 

RM  drives  decisions  on  the  program 

RM  is  done  prior  to  the  program  beginning 

X 

RM  is  done  prior  and  during  program  execution 

RM  is  only  done  during  the  program  execution 

X 

RM  is  done  prior  and  during  program  execution 

risks  are  generalized  through  the  whole  program 

risks  are  categorized 

risk  management  is  done  internally,  only 

an  outside  organization  also  contributes  to  the  RM  process 

risk  is  a  management  function 

risk  is  a  program  team  function 

risks  are  precisely  articulated 

risks  are  generalized,  if  at  all 

each  risk  has  a  consequence 

X 

consequences  are  generalized,  if  at  all 

a  mitigation  strategy  is  completed  for  each  risk 

X 

mitigation  strategy  is  generalized,  if  at  all 

contingency  plans  are  developed  for  a  RM  plan 

X 

contingency  plans  are  ad  hoc  as  problems  arise  in  the  program 

risks  are  anticipated 

X 

if  problems  arise,  management  will  deal  with  it 

the  program  doesn't  have  any  risk 

programs  that  do  not  have  risk,  have  problems 

risk  management  is  automated 

risk  management  may  use  tools,  but  depend  on  human  input 

risks  are  assigned  probabilities 

X 

probabilities  are  not  relevant  for  RM 

all  risks  are  potential  problems,  relative  priorities  for  risks  are  not  useful 

risks  are  weighed  relative  to  other  program  risks  and  thus  prioritized 

risk  management  information  is  only  shared  internally 

X 

risk  management  information  is  shared  with  all  stakeholders 

risk  analysis  uses  ordinal  rankings 

risk  analysis  uses  actual  measurements  with  a  mathematical  model 

regret  analysis  used 

X 

no  regret  analysis  done 

attach  probabilities  to  future  events 

X 

no  probabilities  associated  with  future  events 

assessing  risks  with  mechanical  meethods 

risks  should  be  compared  to  other  risks  and  sorted 

risk  status  tracked 

X 

not  tracked 

technical  risks  examined 

X 

no  technical  risks  examined 

process  risks  examined 

X 

no  process  risks  examined 

product  risks  examined 

X 

no  product  risks  examined 

stakeholder/user  risks  examined 

X 

no  examination  of  stakeholder/user  risks 

checklists  used  to  identify  risks 

X 

no  checklists  used 

risks  are  tracked 

X 

no  tracking  or  monitoring  of  risks 

each  risk  has  an  impact 

X 

no  impact  analysis  of  risk 

each  risk  has  a  mitigation  plan 

no  individual  risk  mitigation 

risks  monitored  by  priority 

X 

no  special  attention  to  track  higher  priority  risks 

risk  assessment  is  formalized 

X 

no  formal  risk  assessment 

risk  control  is  formalized 

X 

no  formal  risk  control 

integration  risks  not  considered 

X 

integration  risks  examined 

Risk  Management  (page  1  of  2)  score  i 

28 
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Pair  choice  section  FOUR:  (Risk  Management(RM))  choose  most  applicable  term  of  the  two  for  each  row  (page  2  of  2): 


risks  to  cost 

X 

no  cost  risks  examined 

unforeseen  risks  have  occurred  in  program 

X 

any  risk  that  came  up  had  been  identified  previously 

personnel  risks  examined 

no  personnel  risks  examined 

X 

estimation  risks  examined 

X 

no  estimation  risks  examined 

planning  risks  examined 

X 

no  planning  risks  examined 

requirements  risks  examined 

X 

no  requirements  risks  examined 

resource  risks  examined 

X 

no  resource  risks  examined 

risk  management  plan  updated  regularly 

X 

no  regular  risk  management  plan  updates 

risks  charted 

X 

risks  not  charted 

performance  risks  examined 

X 

performance  risks  not  examined 

program  management  self  risks  examined 

X 

no  program  management  risks  examined 

risk  from  program  constraints  examined 

X 

no  program  constraint  risks  examined 

each  category  of  risks  are  prioritized 

X 

no  prioritization 

each  category  of  risks  are  evaluated  for  impact 

X 

no  impact  analysis  performed 

each  category  of  risks  have  control  strategy 

X 

no  control  strategy 

documentation  risks  examined 

X 

no  documentation  risks  examined 

regret  matrix  tracked 

X 

no  regret  matrix  or  not  tracked 

communication  of  risk  activities  are  facilitated 

X 

no  facilitation  or  promotion  of  communication  of  risk  activities 

taxonomy-based  questionnaire  used  to  identify  risks 

X 

taxonomy-based  questionnaire  not  used 

associated  hardware  risks  examined 

no  consideration  for  hardware  risks 

X 

integration  risks  examined 

X 

integration  risks  not  examined 

communication  risks  examined 

X 

communication  risks  not  examined 

leadership  risks  examined 

X 

leadership  risks  not  considered 

risk  avoidance  considered  for  certain  risks 

X 

risk  avoidance  not  considered  for  risks 

risk  documentation  forms  used 

X 

no  risk  documentation  forms  used 

dependency  risks  examined 

X 

no  dependency  risks  examined 

alternatives  like  risk  avoidance  considered  for  high  risk  items 

no  consideration  of  risk  avoidance 

X 

documented  risk  statements  use  a  condition-consequence  type  format 

X 

condition-consequence  of  risk  statements  not  clearly  defined 

no  assignment  of  ownership  of  risk  mitigation  action 

each  risk  mitigation  action  is  assigned  to  an  individual  for  resolution 

X 

calculation  of  risk  exposure  made  (probability  X  loss,  for  each  risk) 

X 

no  risk  exposure  calculations 

oral  communication  of  risks  only 

X 

risks  written  in  a  way  that  communicates  nature  and  status  of  factors 

X 

triggers  used  to  quantify  risk  conditions  present 

risk  conditions  present  are  all  subjective 

X 

risk  "czar"  in  program  for  monitoring  risks 

X 

no  special  positions/responsibilities  for  risk  monitoring 

post-program  review  completed  (scheduled)  for  unanticipated  problems  ID 

X 

no  post-program  reviews  completed  or  scheduled 

no  schedule  risks  examined 

risks  to  schedule  investigated 

X 

Risk  Management  (pg  2  of  2)  score  [28]  +pg  1  score  [33]  =  TOTAL  SCORE  [61]  Enter  on  QMM  scoresheet  blk  d. 
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G.  SCORING 

1.  Requirements  Management  Questionnaire 


No.  Requirements  Management  Questionnaire  Yes  No  N/A 


1 

PM  chose  to  have  a  formal  requirements  list 

1 

0 

0 

2 

Requirements  recorded  in  some  way 

2 

-1 

0 

m 

Written  requirements  were  part  of  some  formal  document 

1 

0 

0 

4 

Written  requirements  were  informal 

1 

2 

0 

D 

At  least  some  requirements  were  oral  only 

El 

1 

0 

6 

All  stakeholders  were  identified 

2 

-1 

0 

7 

All  stakeholders  participated  in  the  requirements  extraction 

2 

0 

0 

8 

Some  stakeholders  participated  in  the  requirements  extraction 

1 

0 

0 

mm 

Management  extracted  requirements,  no  stakeholder  involvement 

1 

2 

1 

m 

Management  passed  requirements  to  development  team 

1 

0 

0 

BID 

Stakeholders  not  involvved  in  Management  extraction,  but  approved 

-1 

0 

0 

m 

Management  gets  inputs  from  stakeholders,  then  develops  requirements 

1 

0 

1 

m 

Developers  work  informally  with  users  to  arrive  at  requirements 

1 

0 

0 

m 

Same  as  13,  but  management  oversees  and  formalizes 

MM 

0 

0 

|  If  a  waterfall  or  sequential  development  strategy:  | 

ed 

All  requirements  complete  before  design 

i 

El 

0 

ID 

Some  requirements  left  incomplete  prior  to  design 

-i 

0 

0 

S3 

Requirements  informal  prior  to  design  effort 

-i 

0 

0 

m 

Requirements  serve  as  input 

i 

-1 

0 

KT71 

Length  of  time  for  requirements  work  greater  than  development  work 

2 

-1 

0 

Requirements  developed  in  parallel  to  design 

-1 

1 

0 

|  OR  If  a  prototype,  throwaway,  or  other  development  strategy:  [ 

m 

Learn  about  requirements  through  development  efforts 

1 

-1 

0 

ID 

No  coding  until  all  requirements  are  defined 

El 

1 

0 

ia 

Requirements  formal  prior  to  design  effort 

-i 

0 

0 

ID 

Requirements  serve  as  output 

i 

-1 

0 

KE1 

Requirements  definition  work  in  parallel  to  development  efforts 

2 

-1 

0 

El 

Requirements  developed  in  parallel  to  design 

1 

El 

0 

El 

Are  requirements  frozen  at  some  phase 

1 

El 

0 

W?. 1 

Change  management  exists 

El 

mt 

0 

Change  management  is  formal 

i 

0 

0 

El 

Project  strategy  is  consistent  throughout  development 

i 

0 

0 

E 

Requirements  are  updated 

i 

0 

0 

rea 

Configuration  Management  (CM)  exists 

El 

El 

0 

m 

CM  is  formal 

0 

0 

m. 

Requirements  are  testable 

mm 

El 

0 

El 

Requirements  testing  considered/implemented  during  extraction 

wm 

0 

0 

Bc!il 

Requirements  testing  plan  exists 

MM 

0 

0 

m 

Requirements  testing  is  formal 

0 

0 

Kg 

All  requirements  have  priorities 

K9 

MM 

0 

All  requirements  must  be  implemented 

0 

1 

0 

El 

Requirements  are  tested 

1 

El 

0 

S3 

All  requirements  are  equally  important 

0 

1 

0 

At  least  some  requirements  have  priorities 

1 

0 

0 

ED 

All  requirements  are  traceable 

1 

0 

0 

S3 

Traceability  not  important 

0 

1 

0 

Each  requirement  has  an  author 

1 

0 

0 

E| 

Who  authored  requirement  is  not  important 

0 

1 

0 

El 

Initial  set  of  requirements  to  be  implemented,  no  requirements  creep 

0 

1 

0 

ED 

Structured  and  tracked  changes  to  requirements  only 

1 

El 

0 

ED 

Change  is  inevitable,  changes  allowed  at  all  times 

El 

1 

0 

El 

Change  is  inevitable,  but  changes  limited 

1 

0 

0 

ED 

Requirements  control  funding 

1 

0 

0 

ED 

Requirements  history  kept 

1 

El 

0 

ED 

Baseline  established  for  requirements  at  some  point  prior  to  develop 

MM 

MM 

0 

TOTAL  SCORING 
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2.  Estimation/Planning  Questionnaire 


No.  Estimation/Planning  Questionnaire _ Yes  No  N/A 


1 

A  volume  product  metric  used  (LOC,  #  of  files,  #  of  screens,  pages  of  doc) 

1 

0 

0 

2 

Measure  used  for  various  product  elements  (modules,  components,  CSCI) 

1 

0 

0 

3 

Product  measures  made  by  phase  (amt  at  implementation,  LOC  changed  at  unit  test) 

1 

0 

0 

4 

Other  product  attributes  measured  (FP,  throughput,  mem  cap,  cyclomatic  complexity) 

1 

0 

0 

5 

Product  matrics  tracked  and  updated  hroughout  program  execution 

2 

-1 

0 

6 

Event  count  process  metric  used  (#  defects  in  test,  reqmt  changes,  milestones  met) 

1 

0 

0 

7 

Time  measure  process  metric  used  (cycle  time) 

1 

0 

0 

8 

Process  metrics  tracked  and  updated  throughout  program  execution 

2 

-1 

0 

9 

Program  cost  estimations  made  from  product  or  process  metrics 

1 

0 

0 

10 

Program  cost  extimations  tracked  and  updated  to  reflect  progress/changes 

1 

0 

0 

11 

Factor  analysis  performed  on  program 

1 

0 

0 

12 

Program's  primary  purpose,  including  major  functions  and  deliverables  known 

2 

-1 

0 

13 

Work  breakdown  structure  developed 

2 

-1 

0 

14 

Task  estimated  with  realistic  expectations  of  productivity  probabilities 

1 

-1 

0 

15 

Schedules  developed  based  on  realistic  expectations 

1 

-1 

0 

16 

Schedules  tracked  and  updated  based  on  new  information 

1 

-1 

0 

17 

Detailed  activity  lists  used  for  clearly  defined  completed/not  completed  tasks 

1 

-1 

0 

18 

Quality  assurance  plan  or  similar  to  aid  in  detecting  defects  early  in  program 

1 

-1 

0 

19 

COCOMO  estimates  performed 

1 

-1 

0 

20 

CSCI  clearly  defined  and  tasked 

2 

-1 

0 

21 

Estimates  completed  ad  hoc 

-2 

0 

0 

22 

Gantt  charts  used  and  updated 

1 

-1 

0 

23 

Resource  estimations  (working  hrs,  job  categories,  task  activities)  done 

1 

-1 

0 

24 

Earned  value  established 

2 

-1 

0 

25 

Earned  value  tracked  throughout  program 

2 

0 

0 

26 

Quality  expectations  established  for  product  with  users  and  stakeholders 

1 

-1 

0 

27 

Critical  path  for  program  tasks  developed  and  tracked 

2 

-1 

0 

28 

Measure  of  effectiveness  (MOE)  or  Figure  of  merit  established  and  tracked 

1 

0 

0 

29 

Estimates  are  updated  routinely 

2 

-1 

0 

30 

Schedules  are  updated  routinely 

2 

-1 

0 

31 

Estimations  are  made  by  program  management  (top-down) 

1 

0 

0 

32 

Estimateions  are  made  by  program  team  members  (bottom-up) 

2 

0 

0 

33 

Automated  program  tracking  used 

1 

0 

0 

34 

PM  usually  thorough  in  tracking  and  reporting  schedules  and  financials 

1 

-1 

0 

35 

WBS  developed  only  as  data  call 

-1 

0 

0 

36 

Earned  value  used  to  track  program  progress 

2 

-1 

0 

37 

PM  insists  on  prioritizing  work  reduction  as  schedule/funding  compromised  by 
stakeholders 

1 

-1 

0 

38 

Estimations  are  done  using  both  top  down  and  bottoms  up  approaches 

2 

-1 

0 

39 

All  program  team  members  involved  in  planning  process 

2 

-1 

0 

40 

Plardware  also  considered  in  estimaation  process 

1 

-1 

0 

41 

Program  history  compiled 

1 

0 

0 

42 

System  upgrades  (SCR)  software  changes  requests  estimated  individually 

1 

-1 

0 

43 

Management  duties  apart  of  each  team  member's  responsibilities 

-1 

1 

0 

44 

PM  dictates  schedules  to  program  team 

-1 

0 

0 

45 

Code  reviews  planned  in  schedule 

1 

-1 

0 

46 

Defined  tangible  milestones  established  for  program  tasks 

2 

-1 

0 

47 

Test  planning  done  at  the  start  of  the  program 

1 

-1 

0 

48 

Estimations  are  completed  by  those  performing  the  tasks 

1 

-1 

0 

49 

Sensitivity  analysis  performed  for  program  choices 

1 

-1 

0 

50 

Software  deployment  planning  completed 

1 

-1 

0 

'  =  TOTAL  SCORING 
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3.  People  Management  Questionnaire 


No.  People  Management  Questionnaire _ Yes  No  N/A 


1 

PM  is  accessible  in  person  by  each  team  member 

1 

0 

0 

2 

PM  is  accessible  via  email  (memo,  letter)  by  each  team  member 

1 

0 

0 

3 

PM  is  accessible  via  phone  by  each  team  member 

1 

0 

0 

4 

PM  not  only  considers  a  person's  suitability,  not  also  desire  to  be  on  a  team 

1 

0 

0 

5 

PM  consults  with  each  team  member  regarding  their  career  goals 

1 

0 

0 

6 

PM  regularly  holds  meetings  to  inform  team  of  program  progress 

2 

-1 

0 

7 

PM  solicits  opinions  from  team  members  before  making  decisions 

2 

-1 

0 

8 

PM  lets  teams  make  decisions  affecting  their  work 

1 

0 

0 

9 

PM  freuently  makes  decisions  without  any  consultation  with  members 

-2 

2 

0 

10 

PM  understands  the  technology/language  of  the  program 

1 

0 

0 

11 

PM  is  able  to  communicate  with  other  the  technical  issues  in  the  program 

1 

-1 

0 

12 

PM  prioritized  problems  or  conflicts  within  the  program 

1 

0 

0 

13 

PM  assists  team  members  in  developing/advising  of  career  path 

1 

-1 

0 

14 

PM  empowers  program  members  to  recommend  hiring  new  team  members 

1 

-1 

0 

15 

PM  empowers  program  members  to  recommend  firings  of  other  members 

1 

-1 

0 

16 

PM  specifically  assigns  work  to  each  program  member 

1 

-1 

0 

17 

PM  sets  communication  protocol 

1 

0 

0 

18 

PM  allows  unrestricted  communications 

1 

0 

0 

19 

PM  encourages  or  requires  training  for  each  individual 

1 

-1 

0 

20 

PM  takes  control  in  difficult/roblem  areas 

1 

0 

0 

21 

PM  looks  ahead  to  new  programs,  new  upgrades  of  existing  program 

1 

0 

0 

22 

PM  maintains  regular  communications  with  all  stakeholders 

2 

-1 

0 

23 

PM  maintains  regular  communications  with  users 

2 

-1 

0 

24 

PM  encourages  program  team  communication  with  users 

1 

-1 

0 

25 

PM  encourages  program  team  communication  with  stakeholders 

1 

-1 

0 

26 

PM  facilitates  horizontal  communication  within  program 

1 

-1 

0 

27 

PM  facilitates  communication  during  integration 

1 

-1 

0 

28 

PM  holds  meetings  without  clear  objectives 

-1 

2 

0 

29 

PM  must  approve  all  decisions  within  the  program 

-1 

1 

0 

30 

PM  must  approve  all  interactions  with  stakeholders 

-1 

1 

0 

31 

PM  must  approve  all  interactions  with  users 

-1 

1 

0 

32 

PM  makes  all  presentations  to  stakeholders/users 

0 

1 

0 

33 

PM  is  considered  "flexible"  in  terms  of  program  members  personal  issues 

1 

0 

0 

34 

PM,  at  least  occasionally,  schedules/promotes  outside  work  team  activities 

1 

0 

0 

35 

PM  is  readily  willing  to  listen  to  program  prblems  and  complaints 

1 

-1 

0 

36 

PM  takes  action  to  resolve  program  problems  and  complaints 

1 

-1 

0 

37 

PM  is  generally  respected  by  stakeholders,  users,  and  organization 

1 

-1 

0 

38 

PM  sometimes  fails  to  grasp  important  technical  issues  in  program 

-1 

1 

0 

39 

PM  recruits  program  team  members  from  outside  organization 

1 

-1 

0 

40 

PM  participates  in  technical  reviews 

-1 

1 

0 

41 

Program  personnel  have  clearly  defined  specific  tasks 

0 

1 

0 

42 

Although  individual's  tasks  are  specific,  each  exposed  to  the  "bigger  picture" 

2 

-1 

0 

43 

PM  has  clearly  defined  his/her  expectations  for  each  individual 

2 

-1 

0 

44 

PM  delegation  of  duties  is  usually  seemless  in  execution 

1 

0 

0 

45 

PM  acts  as  facilitator  to  solving  personnel  conflicts 

2 

-1 

0 

46 

PM  attempts  to  motivate  individuals  on  the  program  team 

2 

-1 

0 

47 

PM  clearly  spearates  technical  from  managerial  roles  for  individuals 

0 

1 

0 

48 

PM  directs  how  he/she  expects  the  task  to  be  accomplished 

0 

1 

0 

49 

PM  directs  what  needs  to  be  done,  but  does  not  direct  how 

2 

-1 

0 

50 

PM  attempts  to  spotlight  individuals  in  the  program  for  positive  exposure 

2 

-1 

0 

TOTAL  SCORING 
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4. 


Risk  Management  Questionnaire 


No.  Risk  Management  Questionnaire _ Yes  No  N/A 


1 

Risk  Management  (RM)  is  specifically  an  activity  in  the  program 

4 

-4 

0 

2 

RM  is  formal  and  documented 

3 

-3 

0 

3 

A  specific  RM  Ian  exists 

2 

-2 

0 

4 

RM  is  required  in  the  program,  but  not  used  during  the  program 

-1 

1 

0 

5 

RM  is  done  prior  to  the  program  execution 

1 

0 

0 

6 

RM  is  done  by  an  outside  entity  to  the  development 

1 

0 

0 

7 

RM  is  done  internally  only 

0 

1 

0 

8 

RM  is  both  internally  performed  and  externally  assessed 

1 

-1 

0 

9 

RM  planning  occurs  during  or  after  major  milestones  in  the  program 

1 

-1 

0 

10 

Risk  Assessment  is  only  a  management  function 

0 

1 

0 

11 

RM  is  informal  or  non  existent 

-1 

1 

0 

12 

There  is  a  RM  plan,  but  it  is  not  updated  or  tracked 

1 

0 

0 

13 

Risks  are  only  generalized 

-1 

0 

0 

14 

Each  risk  is  delineated 

1 

0 

0 

15 

Each  risk  has  a  consequence 

1 

0 

0 

16 

Each  risk  has  a  likelihood  rating  of  some  sort 

1 

0 

0 

17 

Each  risk  has  a  mitigation  strategy 

1 

0 

0 

18 

Risk  Management  is  automated 

1 

0 

0 

19 

Risks  are  tracked 

2 

-2 

0 

20 

21 

Regret  analysis  performed 

2 

0 

0 

22 

RM  drives  decisions  in  the  program 

3 

-2 

0 

23 

Risks  have  probabilities 

1 

0 

0 

24 

Risk  Management  is  ad  hoc 

-3 

0 

0 

25 

RM  information  is  shared  with  all  stakeholders  (as  appropriate) 

1 

0 

0 

26 

Risks  are  weighed  relative  to  other  program  risks 

1 

0 

0 

27 

Risk  Assessment  is  a  program  team  activity 

1 

0 

0 

28 

Risk  Assessment  done  prior  to  program  start 

2 

-1 

0 

29 

Risk  Assessment  includes  personnal  risk 

1 

-1 

0 

30 

RM  uses  tools,  but  depends  on  human  decisions 

2 

-1 

0 

31 

Risk  assessment  includes  cost  risks 

1 

0 

0 

32 

Risk  Assessment  includes  schedule  risks 

1 

0 

0 

33 

Risk  Assessment  includes  technology  risks 

1 

-1 

0 

34 

Risk  Assessment  is  briefed  organization  structure  above  program  manager 

1 

-1 

0 

35 

Risk  Assessment  includes  requirements  risks 

1 

-1 

0 

36 

Risk  Assessment  includes  user  risks  (too  little  involvement  of  user) 

1 

0 

0 

37 

Risk  Assessment  includes  documentation  risks 

1 

0 

0 

38 

Risk  Assessment  includes  integration  risks 

1 

-1 

0 

39 

Risk  Assessment  includes  interface  risks  (non-standard) 

1 

-1 

0 

40 

Risk  Assessment  includes  continuing  requirements  change  (feature  creep) 

1 

-1 

0 

41 

Risk  Assessment  includes  dependent  projects/programs  risks 

1 

0 

0 

42 

Documentation  proof  exists  to  demonstrate  following  risk  management  plan 

1 

0 

0 

43 

High  rish  have  measured  tracking  (high  profile  status) 

1 

0 

0 

44 

Organizational  history  used  to  search  for  risks 

1 

0 

0 

45 

Other  organizational  checklists  used  for  risk  assessment 

1 

0 

0 

46 

Internal  organizational  checklists  used  for  risk  assessment 

1 

0 

0 

47 

Risk  Assessment  information  contributed  to  internal  or  other  database 

1 

0 

0 

48 

Risk  Assessment  includes  internal  organization  risks 

1 

0 

0 

49 

Risk  Assessment  includes  stakeholder  risks 

2 

-1 

0 

50 

No  risk  management  needed;  program  is  straightforward  &  understood 

-3 

3 

0 

TOTAL  SCORING 
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5, 


Pair  Choice 


Pair  choice  section  ONE:  (Requirements  Management)  choose  most  applicable  term  of  the  two  for  each  row  (page  1  of  2): 


formal  requirement  list 

2 

informal  requirement  list 

1 

written  requirements 

2 

oral  requirements 

0 

requirements  informal,  but  recorded 

1 

requirements  not  recorded 

0 

requirements  as  part  of  an  SRS  (or  other  formal  repository) 

2 

requirements  informally  recorded 

1 

requirements  taken  as  is  from  customer 

0 

look  to  reformulate,  interview  in-depth,  or  otherwise  re-validate 

2 

only  one  development  strategy  used 

1 

strategies  not  consistent,  used  at  different  times 

0 

stakeholders  as  part  of  requirements  development 

2 

stakeholders  approving  requirements  after  formulated  by  development  team 

1 

requirements  are  testable 

2 

requirements  have  no  test  plans 

0 

informal  test  plan  or  no  test  plan 

0 

formal  test  plan 

2 

test  team  involved  with  requirements 

1 

no  test  team  input  or  plans  during  requirements  development 

0 

only  a  percentage  of  requirements  present  in  baseline 

0 

baseline  must  contain  all  requirements 

2 

requirements  documentation  has  hierarchical  structure 

1 

all  requirements  must  be  implemented 

0 

requirements  have  listed  responsible  party 

1 

requirements  origin  not  important 

0 

requirements  documentation  have  versions 

2 

no  requirements  history 

0 

requirements  have  specific  attribute  values 

1 

requirements  ail  rank  evenly 

0 

funding  controls  requirements  definition 

0 

requirements  definition  controls  funding 

1 

reqquirements  are  top  down 

1 

requirements  are  bottom  up 

2 

users/stakeholders  are  identified  and  interviewed  (market  survey) 

1 

no  special  consideration  to  identify  users/stakeholders 

0 

each  requirement  has  a  singular  concept 

3 

some  requirements  are  compound  statements 

0 

requirements  definition  minimized  when  funding  short 

0 

program  scope  may  reduce,  but  requirements  definition  completed 

1 

requirements  extraction  has  formal  process 

1 

requirements  extraction  ad  hoc 

0 

change  procedures  formal 

1 

change  procedures  ad  hoc 

0 

users/stakeholders  somehow  involved  in  requirements  definition 

1 

program  team  only  involved  in  requirement  definition 

0 

management  sets  requirements  for  developers 

0 

developers  at  least  partially  involved  in  setting  requirements 

1 

requirements  changed  at  least  once  since  baseline  established  prior  to  new  version 

0 

requirements  in  baseline  has  not  changed  prior  to  new  version  or  upgrade 

1 

no  ranking  of  requirements 

0 

requirements  have  priorities  assigned 

1 

use-case  diagrams  (or  other  models  or  scenario  developments) 

2 

no  models  used  for  requirements  extraction 

0 

requirements  changes  informal 

0 

requirements  changes  formal 

1 

plan  to  "freeze"  requirements  at  some  designated  milestone 

1 

no  provision  for  "freezing"  requirements 

0 

requirements  must  be  traceable 

1 

origin  of  requirements  not  important 

0 

requirements  must  be  testable 

3 

system  developed  must  be  testable 

1 

test  plans  to  determine  requirements  implemented 

2 

no  test  plans  needed  for  requirements  verification 

0 

requirements  have  priorities  in  implementation 

1 

all  requirements  must  be  implemented 

0 

some  requirements  have  multiple  statements  or  ideas 

0 

one  idea,  one  statement  per  requirement 

2 

Requirements  Management  (page  1  of  2)  score 
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Pair  choice  section  ONE:  (Requirements  Management)  choose  most  applicable  term  of  the  two  for  each  row  (page  2  of  2): 


|  ANSWER  THIS  BLOCK  OF  QUESTIONS  ONLY  IF  A  SEQUENTIAL  OR  WA  TERFALL  APPROACH  IS  USED  FOR  DEVELOPMENT  (Requirements  page  2  of  2)  || 

requirements  first,  then  initial  development  work 

1 

initial  development  work  then  requirements 

0 

requirements  documentation  driving  development 

1 

requirements  documentation  developed  in  parallel/after  development 

0 

user  feedback  considered  during  development 

1 

after  development  starts,  user  feedback  serves  as  input  to  new  work 

0 

change  management  procedures  used  strictly 

1 

change  management  procedures  as  guidance  only 

0 

design  decisions  prior  to  or  in  parallel  to  requirrements  development 

0 

design  decisions  only  after  approved  requirements  stabilized 

1 

requirements  summarized  wht  we  have  developed 

0 

requirements  are  the  blueprint  for  development 

1 

length  of  time  for  requirements  work  greater  than  development  work 

2 

length  of  time  for  requirements  work  less  than  development  work 

0 

requirements  have  design  detail 

0 

no  design  detail  in  requirements 

i  ! 

requirements  creep  to  be  avoided 

1 

requirements  creep  o.k.,  but  need  to  be  controlled 

0 

freeze  requirements  at  some  point 

1 

requirements  are  fluid  throughout  development 

0 

formal  change  procedure 

1 

informal  change  procedure 

0 

change  management  plan 

2 

no  change  management  plan 

0 

requirements  ambiguity  always  present  to  some  extent 

0 

requirements  ambuiguity  unacceptable  at  any  level 

2 

testing  considered  up  frornt  during  requirements  determination 

2 

testing  considered  down  the  line  during  development 

1 

requirements  development  team  members  different  from  implementation 

0 

those  working  on  requirements,  work  on  implementation 

1 

start  implementation  as  early  as  possible  to  help  define  requirements 

0 

requirements  must  be  defined  prior  to  any  implementation  work 

2 

|  ANSWER  THIS  BLOCK  OF  QUESTIONS  ONLY  IF  A  PROTOTYPING,  THROWAWAY,  SYNCHRONIZE  &  STABILIZE,  OR  OTHER  STRATEGY  USED  | 

develop  prototype,  then  determine  requirements 

1 

determine  requirements  prior  to  any  development  work 

0 

requirements  testing  done  after  each  iteration 

1 

no  testing 

0 

individual  changes  as  necessary 

1 

only  block  changes  made 

0 

development  team  decides  on  changes  after  iteration 

0 

users  involved  with  changes 

1 

changes  based  on  feedback  only  from  user  for  correction  of  problems 

1 

changes  to  upgrade  system  and  correct  problems 

1  ! 

funding  controls  changes  and  change  procedures 

1 

changes  control  funding 

1 

requirements  documentation  finalized  prior  to  development 

0 

requirements  fluid  throughout  development  (only  freeze  at  end) 

2 

requirements  test  plans  completed  prior  to  development 

1 

requirements  test  plans  completed  after  development 

0 

requirements  first,  then  initial  development  work 

0 

initial  development  work  then  requirements 

1 

use  development  effort  to  learn  more  about  requirements 

2 

define  all  requirements  prior  to  coding  anything 

0 

requirements  ambiguity  always  present  to  some  extent 

1 

requirements  ambiguity  unacceptable  at  any  level 

0 

requirements  have  design  detail 

1 

no  design  detail  in  requirements 

1 

user  feedback  considered  during  development 

1 

after  development  starts,  user  feedback  serves  as  input  to  new  work 

0 

get  something  to  users  as  soon  as  possible  for  evaluation 

2 

make  sure  it  is  complete  before  releasing 

0 

management  dictates  requirements 

0 

development  team  visually  represent  requirements  through  rapid  prototyping 

1 

new  requirements  allowed  after  initial  requirements  defined 

1 

new  requirements  not  allowed 

0 

Requirements  Management  (pg  2  of  2)  score 
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Pair  choice  section  TWO:  (Estimation/Planning  Management)  choose  most  applicable  term  of  the  two  for  each  row  (page  1  of  2): 


at  least  one  estimation  method  used  in  program 

1 

no  estimates 

0 

formal  derivation  of  product  metric  for  estimation  of  size 

1 

ad  hoc  size  estimation 

0 

ad  hoc  process  evaluation 

0 

formal  derivation  of  at  lest  one  process  metric 

1 

develop  work  breakdown  structure  (WBS) 

1 

assign  work  as  needs  arise 

0 

estimates  are  developed  to  fulfill  a  data  call  only 

0 

use  estimates  to  plan  program 

1 

use  estimates  to  sell  program  only 

0 

estimates  are  useful  to  the  project  tema  for  planning  purposes 

1 

resource  evaluations  made  for  program 

1 

no  resource  evaluation  for  planning 

0 

use  both  bottom  up  &  top  down  for  estimate,  use  one  stakeholders  like 

0 

use  both  bottom  up  &  top  down  and  evaluate  significant  differences 

1 

estimates  made  and  not  updated 

0 

estimates  updated  throughout  program 

1 

resources  estimations  used  to  adjust  product  size  estimate 

1 

estimations  made  irregardless  of  resources  available 

0 

estimations  made  to  fit  budget 

0 

budget  made  from  estimations 

1 

estimations  compromised  to  get  program 

0 

rather  risk  loss  of  program  than  compromise  confident  estimations 

1 

cycle  time  estimations 

1 

no  cycle  time  estimations 

0 

event  count  estimations 

1 

no  event  count  estimations 

0 

lines  of  code  (LOC)  estimation 

1 

no  LOC  estimation 

0 

function  pont  (FP)  estimation 

1 

no  FP  estimation 

0 

estimates  by  algorithmic  methods 

1 

estimates  by  analogy 

1 

expert  judgement  for  estimates 

1 

ad  hoc  estimates 

0 

estimates  by  algorithmic  methods 

1 

ad  hoc  estimates 

0 

expert  judgement  for  estimates 

0 

estimates  by  analogy 

1 

ad  hoc  estimates 

0 

estimates  by  analogy 

1 

bottom  up  estimates 

1 

expert  judgement 

0 

top  down  estimates 

1 

expert  judgement 

0 

ad  hoc  estimates 

0 

any  other  estimate  process 

1 

fuzzy  logic  estimating  method 

1 

no  formal  estimation  methodology 

0 

WBS  development  from  estimates 

1 

WBS  development  in  parallel  or  prior  to  estimation  completion 

0 

critical  path  of  program  determined 

1 

tasks  developed  but  no  path  is  identified 

0 

estimators  are  program  team  members 

1 

estimators  are  outside  program  team 

0 

management  only  on  estimations 

0 

ail  team  members  involved  in  estimation  process 

1 

estimates  updated  at  reviews 

1 

no  updates  of  estimates 

0 

estimates  updated  at  reviews 

0 

estimates  constantly  updates  (in  between  reviews,  to) 

1 

estimate  procedures  stay  the  same 

1 

estimate  procedures  change 

0 

stakeholders  are  part  of  estimation  process 

1 

stakeholders  brief  estimations  after  completion 

0 

estimates  are  used  beyond  initial  selling  of  program 

1 

estimates  are  one  time  events,  used  for  a  specific  purpose  once 

0 

WBS  has  objective  measure  of  completeness 

1 

important  to  have  WBS  as  guide,  not  rigid  implementation 

0 
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Pair  choice  section  TWO:  (Estimation/Planning  Management)  choose  most  applicable  term  of  the  two  for  each  row  (page  2  of  2): 


life  cycle  estimates 

1 

estimates  for  program  initiation  only 

0 

system  upgrades  (SCR)  software  change  requests  estimated  individually 

1 

systems  upgrades  estimated  as  whole 

0 

estimates  for  on-gong  resources  needed  to  maintain  s/w 

1 

estimates  for  maintenance  not  done 

0 

informal  re-estimates  during  development 

0 

formal  re-estimates  at  pre-defined  milestones 

1 

formal  re-estimates  when  amendment  changing  the  system  is  introduced 

1 

informal  re-estimates  when  amendment  changing  the  system 

0 

person  in-charge  of  estimation  walks  in  a  managers  office  to  get  an  opinion 

0 

meeting(s)  organized  for  purpose  of  performing  cost  estimations 

1 

factor  analysis  prior  to  commencement  of  program 

1 

none  done 

0 

change  control  procedures  set  in  place 

1 

no  set  procedures 

0 

elapsed  time  and  actual  work  time  estimates 

1 

one  or  the  other  or  neither 

0 

no  schedule  created 

0 

scheudle  created 

1 

schedule  not  updated 

0 

schedule  updated 

1 

schedule  followed 

1 

schedule  not  followed 

0 

tasks  identification  arises  as  program  progresses 

0 

detailed  level  tasks  identified  prior  to  program  initiation 

1 

scope  of  program  understood  by  all 

1 

scope  not  explicitly  defined 

0 

quality  factors  and  criteria  identified 

1 

no  explicit  quality  factors  defined 

0 

no  project  tracking  tools  used 

0 

project  tracking  tools  used 

1 

CSCIs  identified  and  tasked 

1 

CSCIs  not  explicitly  identified 

0 

expectations  are  managed  via  estimations 

1 

estimations  are  made  to  fit  preconceived  expectations 

0 

no  cost  schedule  developed 

0 

cost  schedule  developed 

1 

no  resource  schedule  developed 

0 

resource  schedule  developed 

1 

team  members,  management  know  at  any  time  if  in  budget  &  schedule 

1 

exact  budget  &  schedule  status  somewhat  unclear  to  at  least  some 

0 

individual  program  phases  are  estimated 

1 

only  top  level  program  estimated 

0 

stakeholders/users  emphasis  understood-quick  to  field  or  all  complete 

1 

program  management  sets  delivery  tradeoffs  without  outside  input 

0 

testing  planned  with  initial  program  planning 

1 

testing  not  in  initial  planning 

0 

documentation  not  considered  ininitial  planning 

0 

documentation  part  of  initial  planning 

1 

hardware  considered  in  estimations 

1 

software  only  considered 

0 

no  formal  schedule/cost  tracking 

0 

formal  procedures  established  for  tracking  cost  and  schedule 

1 

earned  value  set  up 

1 

earned  value  not  used 

0 

estimations  omit  documentation  planning 

0 

documentation  in  estimates 

1 

training  omitted  in  estimates 

0 

training  part  of  estimates 

1 

earned  value  set  up,  but  not  tracked 

0 

earned  value  tracked 

1 

detailed  planning  done  with  incomplete  set  of  requirements 

0 

detailed  planning  done  with  detailed  set  of  requirements 

1 

complete  infrastructure  support  mechanism  understood  for  estimations 

1 

no  consideration  of  infrastructure  done  for  estimations 

0 

team  possibilities  considered  for  planning  of  program 

1 

no  consideration  for  outside  teaming  possibilities 

0 

work  breakdown  structure  (WBS)  set  up 

1 

no  WBS  completed 

0 
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Pair  choice  section  THREE:  (People  Management)  choose  most  applicable  term  of  the  two  for  each  row  (page  1  of  2): 
Human  Resources 


program  team  members  have  clearly  deined,  segmented  roles 

0 

work  responsibilities  are  shared 

1 

formal  team  building  procedures  are  used 

1 

no  formal  team  building  emphasized 

0 

program  manager  flexible  regarding  work  hours 

1 

program  manager  maintains  strict  standards  for  work  hours 

0 

big  picture  conveyed  to  all  team  members  by  program  management 

1 

program  management  focuses  on  the  partitioned  tasks  with  team 

0 

people  issues  dealt  with  primarily  through  indirect  methods  (email,  memo,  etc) 

0 

people  issues  dealt  with  primarily  through  direct  methods  (face-to-face) 

1 

training  is  required  and  planned  on  a  regular  basis 

1 

training  is  ad  hoc 

0 

each  team  member  is  educated  on  and  understands  overall  program  and  their  role 

1 

team  members  only  know  their  respective  areas 

0 

consideration  for  team  members'  career  goals  are  reflected  in  assignments 

1 

team  members  must  adapt  to  tasks  that  are  assigned 

0 

team  members  assignments  and  responsibilities  are  mostly  dictated  by  PM 

0 

assignments  and  responsibilities  are  discussed  and  agreed  upon  with  PM 

1 

management  leads  in  problem  solving 

0 

management  facilitates  and  lets  team  lead  in  problem  solving 

1 

management  welcomes  problems  as  challenges  and  opportunities 

1 

management  views  problems  as  obstacles  and  grounds  for  punishment 

0 

team  members  participate  in  performance  evaluations  of  peers 

1 

Personnel  evaluations  are  strictly  PM  responsibility 

0 

management  reinforcement  feedback  sparse  and  inconsistent,  if  any 

0 

management  provides  timely  reinforcement  feedback  for  positive  behaviors 

1 

management  provides  basic  needs  of  office  facilities  fairly  well 

1 

office  facilities  are  a  drawback  to  working  in  the  program 

0 

working  conditions  are  fairly  comfortable,  time  off  policy  fairly  good 

1 

working  conditions  and  time  off  policy  is  inconsistent  and  difficult  at  times 

0 

Communication: 


communications  primarily  written  (email) 

1 

communications  primarily  verbal  (face-to-face) 

1 

detailed  instructions:  oral  presentation,  follow-up  email 

1 

email  only 

0 

formal  communication  protocol 

1 

informal  communications 

0 

external  vertical  communications  restricted 

0 

external  vertical  communication  allowed 

1 

coders  notebook  weekly  accomplishment  reports  required 

1 

not  required 

0 

user-coder  relationship  established,  encouraged,  and  mediated 

1 

user-coder  interaction  minimized 

0 

meetings  structured  to  minimize  waster  time 

1 

meetings  unstructured  and  open  ended 

0 

meetings  have  agenda,  objectives,  and  conclude  with  action  items 

1 

meeting  agenda  fluid  and  open  ended 

0 

program  management  and  coder  communication  face  to  face 

1 

program  management  and  coder  communication  primarily  email 

0 

program  team  updated  regularly  regarding  organizational  &  program  status 

1 

meetings  infrequently  scheduled 

0 

open  communications  is  encouraged 

1 

communication  hrough  chain  of  command  only  is  encouraged 

0 

program  manager  accessible  for  discussions 

1 

program  manager  difficult  to  get  an  appointment  to  see 

0 

program  management  (PM)  is  viewed  as  separate  from  team 

0 

PM  mixes  with  team  frequently 

1 

management  regularly  holds  team  meetings 

1 

meetings  are  sporadic 

0 

meetings  are  structured  with  definite  goals  and  objectives 

1 

meetings  are  informal 

0 

program  management  is  generally  easy  to  reach  and  talk  to 

1 

PM  is  usually  hard  to  get  a  hold  of  and  difficult  to  talk  to 

0 

team-program  manager  relationship  adult-adult 

1 

team-program  management  relationship  parent-child 

0 

schedules  are  spontaneous  and  poorly  communicated 

0 

schedules  must  be  fixed  and  rigidly  followed  and  formally  reported 

1 

work  is  seen  as  complex  processes  involving  team  working  together 

1 

work  broken  into  pieces  with  minimal  team  member  interaction 

0 

action  items  often  is  poorly  disseminated  and  usually  not  followed  through 

0 

action  items  communicated  and  followed  through  thoroughly 

1 

team  members  require  frequent  clarifications  by  PM  for  assigned  tasks 

0 

team  members  rarly  require  clarifications  by  PM  for  assigned  tasks 

1 
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Pair  choice  section  THREE:  (People  Management)  choose  most  applicable  term  of  the  two  for  each  row  (page  2  of  2): 
Leadership: 


long  range  organizational  vision 

1 

short  tern  program  and  immediate  work  focus 

0 

lead  through  personal  attention  to  others 

1 

action-oriented  leadership  approach 

1 

run  as  much  of  the  organization  as  possible 

0 

let  team  make  decisions  as  much  as  possible 

1 

direct  and  domineering  style 

0 

encourage  independence  in  others 

1 

traditional  leaders  respect  hierarchy 

0 

do  what  needs  to  be  done 

1 

win  cooperation  rather  than  demand  it 

1 

tough-minded  with  others 

0 

act  strongly  and  forcefully  in  the  field  of  ideas 

0 

prefer  to  lead  other  independent  types  while  seeking  autonomy  for  self 

1 

consults  with  team  members  to  find  solutions  to  problems 

1 

consults  team  members  to  get  validation  of  PM's  predetermined  solutions 

0 

keep  people  well  informed 

1 

only  as  much  knowledge  as  necessary  for  their  work 

0 

make  things  happen  by  focusing  on  the  immediate  problems 

1 

long  range  focus  and  de-emphasize  current  problems 

1 

manage  others  loosely  and  prefer  minimal  supervision 

1 

follow  traditional  procedures  and  rules  conscientiously 

0 

leadership,  management  decisions  exclusively  by  program  management 

0 

program  management  makes  decisions  but  gets  inputs  from  team 

1 

team-program  manager  relationship  adult-adult 

1 

team-program  management  relationship  parent-child 

0 

program  management  makes  decisions  but  gets  inputs  from  team 

0 

all  program  team  members  responsible  for  program  decisions 

1 

when  a  problem  arises:  management  takes  over  to  solve  it 

0 

management  lets  the  team  solve  the  problems 

1 

leadership  is  do  as  1  say,  not  do  as  1  do 

0 

leadership  by  example 

1 

program  expectation  not  influenced  by  PM 

0 

program  expectation  managed  by  PM 

1 

PM  gives  freedom  to  team,  but  has  no  mentoring  for  members  (abdication) 

0 

PM  empowers  teams  by  mentoring  members  to  be  leaders 

1 

promgram  management  waits  and  sees  what  happens  then  plans 

0 

management  plans  far  in  advance 

1 

program  management  is  constantly  reacting  to  emergencies 

0 

management  is  one  step  ahead  of  problems 

1 

facilitative  approach  to  solving  problems 

1 

take  charge  readily  and  often 

0 

program  management  is  complex,  takes  much  time  to  understand 

0 

management  is  simple,  easy  to  figure  out 

1 

program  management  prefers  to  plunge  right  in 

0 

takes  time  to  separate  things  to  be  done  and  order  of  doing  them 

1 

program  management  reacts  spur  of  the  moment 

0 

methodically  follows  plans 

1 

Technical  Competency  of  the  Program  Manager: 


PM  has  technical  experience  particular  to  the  particular  s/w  program 

1 

PM  relies  on  team  members  solely 

0 

PM  participates  in  technical  reviews 

1 

PM  only  in  non-technical  reviews 

0 

PM  participates  in  making  technical  decisions  when  problems  arise 

1 

PM  delegates  technical  questions 

0 

PM  does  not  get  involved  discussing  technical  options 

0 

PM  contributes  to  technical  options  being  discussed 

1 

PM  does  not  review  technical  options  and  decisions 

0 

PM  reviews  technical  options  and  decisions 

1 

PM  actively  attempts  to  keep  up-to-date  with  current  technology  and  standards 

1 

PM  is  removed  from  cutting  edge  technology  issues 

0 

PM  receives  technical  periodicals  and  occasionally  references  applicable  articles 

1 

PM  doesn't  read  periodicals  nor  reference  current  articles  to  team 

0 

PM  doesn't  have  technical  background  (or  education) 

0 

PM  has  technical  background  (or  education) 

1 

team  members  avoid  PM  when  they  need  technical  advice 

0 

team  members  generally  consider  talking  to  PM  regarding  technical  issues 

1 

HR 


+  Comm. 


□  +  Leadership 


+  Tech.  Competency 


=  People  Mgmt.  score 


Enter  on  QMM  scoresheet  blk  c. 
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Pair  choice  section  FOUR:  (Risk  Management(RM))  choose  most  applicable  term  of  the  two  for  each  row  (page  1  of  2): 


RM  is  formal  and  documented 

1 

RM  is  informal,  if  at  all 

0 

a  risk  management  plan  exists 

1 

no  risk  management  plan  is  developed 

0 

RM  is  more  of  a  data  call  than  a  useful  document 

0 

RM  drives  decisions  on  the  program 

1 

RM  is  done  prior  to  the  program  beginning 

0 

RM  is  done  prior  and  during  program  execution 

1 

RM  is  only  done  during  the  program  execution 

0 

RM  is  done  prior  and  during  program  execution 

1 

risks  are  generalized  through  the  whole  program 

0 

risks  are  categorized 

1 

risk  management  is  done  internally,  only 

0 

an  outside  organization  also  contributes  to  the  RM  process 

1 

risk  is  a  management  function 

0 

risk  is  a  program  team  function 

1 

risks  are  precisely  articulated 

1 

risks  are  generalized,  if  at  all 

0 

each  risk  has  a  consequence 

1 

consequences  are  generalized,  if  at  all 

0 

a  mitigation  strategy  is  completed  for  each  risk 

1 

mitigation  strategy  is  generalized,  if  at  all 

0 

contingency  plans  are  developed  for  a  RM  plan 

1 

contingency  plans  are  ad  hoc  as  problems  arise  in  the  program 

0 

risks  are  anticipated 

1 

if  problems  arise,  management  will  deal  with  it 

0 

the  program  doesn't  have  any  risk 

0 

programs  that  do  not  have  risk,  have  problems 

1 

risk  management  is  automated 

0 

risk  management  may  use  tools,  but  depend  on  human  input 

1 

risks  are  assigned  probabilities 

1 

probabilities  are  not  relevant  for  RM 

0 

all  risks  are  potential  problems,  relative  priorities  for  risks  are  not  useful 

0 

risks  are  weighed  relative  to  other  program  risks  and  thus  prioritized 

1 

risk  management  information  is  only  shared  internally 

0 

risk  management  information  is  shared  with  all  stakeholders 

1 

risk  analysis  uses  ordinal  rankings 

0 

risk  analysis  uses  actual  measurements  with  a  mathematical  model 

1 

regret  analysis  used 

1 

no  regret  analysis  done 

0 

attach  probabilities  to  future  events 

1 

no  probabilities  associated  with  future  events 

0 

assessing  risks  with  mechanical  meethods 

0 

risks  should  be  compared  to  other  risks  and  sorted 

1 

risk  status  tracked 

1 

not  tracked 

0 

technical  risks  examined 

1 

no  technical  risks  examined 

0 

process  risks  examined 

1 

no  process  risks  examined 

0 

product  risks  examined 

1 

no  product  risks  examined 

0 

stakeholder/user  risks  examined 

1 

no  examination  of  stakeholder/user  risks 

0 

checklists  used  to  identify  risks 

1 

no  checklists  used 

0 

risks  are  tracked 

1 

no  tracking  or  monitoring  of  risks 

0 

each  risk  has  an  impact 

1 

no  impact  analysis  of  risk 

0 

each  risk  has  a  mitigation  plan 

1 

no  individual  risk  mitigation 

0 

risks  monitored  by  priority 

1 

no  special  attention  to  track  higher  priority  risks 

0 

risk  assessment  is  formalized 

1 

no  formal  risk  assessment 

0 

risk  control  is  formalized 

1 

no  formal  risk  control 

0 

integration  risks  not  considered 

0 

integration  risks  examined 

1 

Risk  Management  (page  1  of  2)  score 
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Pair  choice  section  FOUR:  (Risk  Management(RM))  choose  most  applicable  term  of  the  two  for  each  row  (page  2  of  2): 


risks  to  cost 

1 

no  cost  risks  examined 

0 

unforeseen  risks  have  occurred  in  program 

0 

any  risk  that  came  up  had  been  identified  previously 

1 

personnel  risks  examined 

1 

no  personnel  risks  examined 

0 

estimation  risks  examined 

1 

no  estimation  risks  examined 

0 

planning  risks  examined 

1 

no  planning  risks  examined 

0 

requirements  risks  examined 

1 

no  requirements  risks  examined 

0 

resource  risks  examined 

1 

no  resource  risks  examined 

0 

risk  management  plan  updated  regularly 

1 

no  regular  risk  management  plan  updates 

0 

risks  charted 

1 

risks  not  charted 

0 

performance  risks  examined 

1 

performance  risks  not  examined 

0 

program  management  self  risks  examined 

1 

no  program  management  risks  examined 

0 

risk  from  program  constraints  examined 

1 

no  program  constraint  risks  examined 

0 

each  category  of  risks  are  prioritized 

1 

no  prioritization 

0 

each  category  of  risks  are  evaluated  for  impact 

1 

no  impact  analysis  performed 

0 

each  category  of  risks  have  control  strategy 

1 

no  control  strategy 

0 

documentation  risks  examined 

1 

no  documentation  risks  examined 

0 

regret  matrix  tracked 

1 

no  regret  matrix  or  not  tracked 

0 

communication  of  risk  activities  are  facilitated 

1 

no  facilitation  or  promotion  of  communication  of  risk  activities 

0 

taxonomy-based  questionnaire  used  to  identify  risks 

1 

taxonomy-based  questionnaire  not  used 

0 

associated  hardware  risks  examined 

1 

no  consideration  for  hardware  risks 

0 

integration  risks  examined 

1 

integration  risks  not  examined 

0 

communication  risks  examined 

1 

communication  risks  not  examined 

0 

leadership  risks  examined 

1 

leadership  risks  not  considered 

0 

risk  avoidance  considered  for  certain  risks 

1 

risk  avoidance  not  considered  for  risks 

0 

risk  documentation  forms  used 

1 

no  risk  documentation  forms  used 

0 

dependency  risks  examined 

1 

no  dependency  risks  examined 

0 

alternatives  like  risk  avoidance  considered  for  high  risk  items 

1 

no  consideration  of  risk  avoidance 

0 

documented  risk  statements  use  a  condition-consequence  type  format 

1 

condition-consequence  of  risk  statements  not  clearly  defined 

0 

no  assignment  of  ownership  of  risk  mitigation  action 

0 

each  risk  mitigation  action  is  assigned  to  an  individual  for  resolution 

1 

calculation  of  risk  exposure  made  (probability  X  loss,  for  each  risk) 

1 

no  risk  exposure  calculations 

0 

oral  communication  of  risks  only 

0 

risks  written  in  a  way  that  communicates  nature  and  status  of  factors 

1 

triggers  used  to  quantify  risk  conditions  present 

1 

risk  conditions  present  are  all  subjective 

0 

risk  "czar"  in  program  for  monitoring  risks 

1 

no  special  positions/responsibilities  for  risk  monitoring 

0 

post-program  review  completed  (scheduled)  for  unanticipated  problems  ID 

1 

no  post-program  reviews  completed  or  scheduled 

0 

no  schedule  risks  examined 

0 

risks  to  schedule  investigated 

1 

Risk  Management  (pg  2  of  2)  score 


+pg  1  score 


=  TOTAL  SCORE 


Enter  on  QMM  scoresheet  blk  d. 
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Earned  Value  Management 

‘Gold  Card’ 

5-.; )  Defense  Acquisition  University 


VARIANCES  revcmrie  is  Pcehue,  wrfav  treble  8  Neseb/e 
Cost  Variance  CV  =  BCWP  -  ACWP  CV  %  =  (CV  /  BCWP)  .100 

Sclwlu  4  Vananc*  SV  =  BCWP  -  BCWS  SV  %  =  (SV I BCWS)  •  1 00 

variance  at  Completion  VAC  =  BAC  -  EAC 

PERFORMANCE  INDICES  FeveraMe  is  >  1 0,  wffe/owbte  3  <  1 .0 
Cost  Efficiency  CPI  =  BCWP  /  ACWP 

Scliodu  e  EffioMcy  SPi  =  BCWP  /  BCWS 
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APPENDIX  C 


A 

SPI 

0.98 

0.02 

1.00 

0.00 

1.00 

0.00 

1.21 

0.21 

1.18 

0.18 

1.15 

0.15 

1.00 

0.00 

1.00 

0.00 

1.00 

0.00 

1.00 

0.00 

105 

1.05 

CPI 

1.09 

0.09 

1.07 

0.07 

1.02 

0.01 

1.24 

0.24 

1.14 

0.14 

1.14 

0.14 

1.14 

0.14 

1.14 

0.14 

1.14 

0.14 

1.14 

0.14 

113 

1.13 

B 

SPI 

1.00 

0.00 

1.00 

0.00 

1.00 

0.00 

1.00 

0.00 

1.00 

0.00 

1.00 

0.00 

1.00 

0.00 

1.00 

0.00 

1.00 

0.00 

1.00 

0.00 

100 

1 

CPI 

1.00 

0.00 

1.00 

0.00 

1.00 

0.00 

1.00 

0.00 

1.00 

0.00 

1.00 

0.00 

1.00 

0.00 

1.00 

0.00 

1.00 

0.00 

1.00 

0.00 

100 

1 

C 

SPI 

0.99 

0.02 

0.98 

0.02 

0.98 

0.02 

0.98 

0.02 

0.97 

0.03 

0.97 

0.03 

0.96 

0.04 

0.98 

0.03 

0.98 

0.03 

0.98 

0.03 

97.4 

0.97 

CPI 

0.95 

0.05 

0.96 

0.04 

0.96 

0.04 

0.97 

0.03 

1.01 

0.01 

1.01 

0.00 

1.04 

0.04 

1.05 

0.04 

1.05 

0.05 

1.05 

0.04 

100 

1 

Program 

Program 

Program 

Program 

Participant 

ApM 

Ai 

BpM 

Bi 

CpM 

Ci 

QMM  score 

77 

79 

86 

85 

48 

45 

QMM  percent 

77 

79 

86 

85 

48 

45 

Success  score 

8 

8 

9 

8 

6 

6 

Mean  success 
score  (0-10) 

8 

8.5 

6 
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